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Speaker Bios 


Tom Baker, J.D. 
William Maul Measey Professor of Law and Health Sciences, University of Pennsylvania Law School 


Tom Baker, a preeminent scholar in insurance law, explores insurance, risk, and responsibility using methods 
and perspectives drawn from economics, sociology, psychology, and history. He has conducted wide-ranging 
research on many topics, including property and casualty insurance, life and health insurance, the behavioral 
economics of insurance, tort law in action, and the historical development of insurance ideas and institutions. 
His behavioral decision research has documented the impact of choice architecture and decision support on 
health plan choice, both in the lab and in the marketplace. Current research projects include cyber liability and 
insurance, and the regulation of secondary insurance markets. Professor Baker is the Reporter for the 
American Law Institute’s Restatement of the Law, Liability Insurance, Director of the Health Insurance 
Exchange Research Group of Penn’s Leonard Davis Institute of Health Economics, and co-founder of Picwell, 
a health data analytics company that provides advanced consumer decision support tools to exchanges, 
insurers, and employers. In August 2013 he received the Robert B. McKay award, a lifetime scholarly 
achievement award given by the Tort Trial and Insurance Practice Section of the American Bar Association. 
Professor Baker holds a J.D. from Harvard Law School and a B.A. from Harvard College. 


Solon Barocas, Ph.D. 
Assistant Professor, Department of Information Science, Cornell University 


Solon Barocas is an Assistant Professor in the Department of Information Science at Cornell University. He is 
also a Faculty Associate at the Berkman Klein Center for Internet & Society at Harvard University. His 
current research explores ethical and policy issues in artificial intelligence, particularly fairness in machine 
learning, methods for bringing accountability to automated decision-making, and the privacy implications of 
inference. He co-founded the annual workshop on Fairness, Accountability, and Transparency in Machine 
Learning (FAT/ML) and later established the ACM conference on Fairness, Accountability, and Transparency 
(FAT*). 


D. Paul Benjamin, Ph.D. 
Director, Pace Robotics Lab and Professor of Computer Science, Seidenberg School of CSIS, Pace 
University 


Dr. Benjamin earned his doctorate in computer science from New York University in 1985. He worked for six 
years at Philips Laboratories before entering academia. Currently, he is professor of Computer Science and 
Director of the Robotics Lab at Pace University in New York City. His research focuses on issues of problem 
solving and representation in cognitive architectures. He and his research group are developing cognitive 
agents in cyber security and robotics. 


Jonah Crane, J.D. 
Executive Director, RegTech Lab; Regulator in Residence, FinTech Innovation Lab 


Jonah Crane is a strategic advisor to financial technology startups, helping them stay ahead of regulatory 
change as they scale. Jonah is also Regulator in Residence at the FinTech Innovation Lab in New York, and 
Executive Director of RegTech Lab in Washington, D.C. Jonah previously served as Deputy Assistant 
Secretary and Senior Advisor at the U.S. Treasury Department, and as a policy advisor to Senator Chuck 
Schumer. Before joining Senator Schumer’s staff, Jonah was a corporate attorney focusing on mergers and 


acquisitions at Milbank, Tweed, Hadley & McCloy LLP in New York. Jonah received a J.D. from New York 
University School of Law. 


Rebecca Crootof, J.D., Ph.D. 
Clinical Lecturer in Law and Research Scholar in Law, Executive Director, Information Society Project, 
Yale Law School 


Rebecca Crootof is a Research Scholar and Lecturer in Law at Yale Law School and the Executive Director of 
the Information Society Project. Dr. Crootof’s primary areas of research include torts, international law, 
national security, and cyber and technology law; her written work explores questions stemming from the 
iterative relationship between law and technology, often in light of social changes sparked by increasingly 
autonomous systems, artificial intelligence, cyberspace, robotics, and the Internet of Things. She teaches 
“Technology Law,” a course that identifies the various ways both domestic and international legal regimes 
respond to and shape technological development. Dr. Crootof earned a B.A. cum laude in English with a 
minor in Mathematics at Pomona College; a J.D. at Yale Law School; and a Ph.D. at Yale Law School. She 
served as a law clerk for Judge Mark R. Kravitz of the U.S. District Court for the District of Connecticut and 
for Judge John M. Walker, Jr. of the U.S. Court of Appeals for the Second Circuit. She is a member of the 
New York Bar, serves on the Board of Directors of the Equal Rights Center, and is a member of the Center for 
New American Security’s Task Force on Artificial Intelligence and National Security. She enjoys skiing, 
hiking, and reading science fiction “for work.” 


Deborah W. Denno, Ph.D., J.D. 
Arthur A. McGivney Professor of Law, Founding Director, Neuroscience and Law Center, Fordham 
University School of Law 


Deborah W. Denno is the Arthur A. McGivney Professor of Law and Founding Director of the Neuroscience 
and Law Center at Fordham University School of Law. She received her B.A. from the University of Virginia, 
her M.A. from the University of Toronto, her Ph.D. in sociology with a specialty in criminology from the 
University of Pennsylvania, and her J.D. from the University of Pennsylvania Law School, where she was the 
Managing Editor of the University of Pennsylvania Law Review. Prior to joining the Fordham Law faculty in 
1991, Professor Denno clerked for the Honorable Anthony J. Scirica of the Third Circuit Court of Appeals and 
worked as an associate at Simpson, Thacher & Bartlett LLP. Professor Denno’s Neuroscience and Law Center 
provides evidence-based information to academics, lawyers, and the public about legally relevant advances in 
neuroscience with the goal of fostering legal scholarship and the use of neuroscience in legal circles. Also at 
Fordham Law, Professor Denno teaches criminal law, criminal procedure, torts, and seminars on topics such as 
law and neuroscience as well as advanced criminal law and advanced criminal procedure. In 2016, the 
Fordham Student Bar Association named Professor Denno Teacher of the Year. In 2007, the National Law 
Journal selected Professor Denno as one of its “Fifty Most Influential Women Lawyers in America.” Seven of 
Professor Denno’s articles have been cited by the United States Supreme Court, and the Court has cited four of 
her articles in the last three years (2015-2017). Professor Denno has published on a broad range of topics 
relating to criminal law, criminal procedure, social sciences and the law, and the death penalty, including the 
constitutionality of execution methods. Professor Denno has also initiated cutting-edge examinations of 
criminal law defenses pertaining to insanity, rape law, gender differences, consciousness, biological and 
genetic links to crime, drug offenses, jury decision-making, and the impact of lead poisoning. Currently she is 
working on a book to be published by Oxford University Press analyzing all criminal cases during a two- 
decade period that have addressed neuroscience evidence. This same study is discussed in some of her most 
recent articles, Concocting Criminal Intent, 105 Geo. L.J. 323 (2017) and Neuroscience and the 
Personalization of Criminal Law, 85 U. Chi. L. Rev. _ (2019) (forthcoming). 


Edward W. Felten, Ph.D. 
Robert E. Kahn Professor of Computer Science and Public Affairs, Director, Center for Information 
Technology Policy, Princeton University 


Edward W. Felten is the Robert E. Kahn Professor of Computer Science and Public Affairs Director, Center 
for Information Technology Policy (CITP) at Princeton University. CITP is a cross-disciplinary effort to study 
digital technologies in public life and has seventeen affiliated faculty members and maintains a diverse 
research program and a busy events schedule. Professor Felten often blogs about technology and policy at 
Freedom to Tinker. He received his Ph.D. in Computer Science and Engineering from the University of 
Washington and holds an M.S. in Computer Science and Engineering from the University of Washington and a 
B.S. in Physics from California Institute of Technology. 


Janet Freilich, J.D. 
Associate Professor of Law, Fordham University School of Law 


Janet Freilich is an Associate Professor of Law at Fordham University School of Law School she researches 
and teaches on civil procedure, intellectual property, and patent law. Before coming to Fordham, Professor 
Freilich was Harvard Law School’s inaugural postdoctoral fellow in private law and intellectual property with 
the Program on the Foundations of Private Law and a patent litigator and prosecutor at Covington & Burling 
LLP. She graduated magna cum laude from Harvard Law School and summa cum laude from Cornell 
University with a bachelor’s degree in molecular biology. 


Caroline Gentile, J.D. 
Associate Professor of Law, Fordham University School of Law 


Caroline M. Gentile is an Associate Professor of Law at Fordham University School of Law, where she played 
an integral role in the development of its Corporate Law Center and served, for two years, as the Center’s 
Academic Director. Professor Gentile teaches Corporations (Business Associations), Corporate Finance, 
Mergers and Acquisitions, and Securities Regulation. Her research interests lie in the integration of economic 
analysis, principally theories of the firm and finance theory, into the legal analysis of the organization of 
business enterprises and the operation of the capital markets (including the market for sovereign bonds). Her 
work has been published in the Duke Law Journal, Emory Law Journal, and Journal of Law & Contemporary 
Problems. Professor Gentile received her undergraduate degrees, summa cum laude, in Economics and 
Political Science from The Ohio State University. She studied in the Law and Economics Program at Yale 
University, and she received her law degree from the Yale Law School, where she was Symposium Editor of 
the Yale Law Journal and Editor of the Yale Journal on Regulation. Following her studies, Professor Gentile 
was an associate in the Corporate Department of Cravath, Swaine & Moore LLP, specializing in the areas of 
debt financing, mergers and acquisitions, and securities offerings. From 2000 to 2002, Professor Gentile 
served as a Lecturer in Law at the UCLA School of Law, teaching both traditional business law courses and a 
practicum in Corporate Financing Transactions. For the academic year 2010 to 2011, Professor Gentile served 
as a Visiting Associate Professor of Law and the Ruby R. Vale Research Scholar at the Wilmington Campus of 
the Widener University School of Law. 


Iria Giuffrida, Ph.D., LL.M. 
Associate Director for Research, Center for Legal and Court Technology; Visiting Assistant Professor of 
Law, William & Mary Law School 


Iria is a Visiting Assistant Professor at William & Mary Law School and has been appointed Associate 
Director for Research at the Center for Legal & Court Technology. Over the last two academic years, she has 


been working on a grant-funded research project focusing on legal issues arising from the increasing use of 
artificial intelligence, the rapid growth of the internet of things and related emerging technologies. She also 
co-teaches an innovative law course on the same subject, as well as courses on International Commercial 
Arbitration and International Business Transactions. In her previous professional life, she was a commercial 
litigator with Dechert LLP (in the London and Brussels offices) and for Enyo Law LLP, a big-ticket specialist 
litigation firm in London. She gained substantial experience in alternative dispute resolution having been 
involved in a number of ICC and LCIA arbitrations as well as mediations, and she acted on complex, 
multijurisdictional financial services disputes. Prior to practicing law, Iria taught law in the U.S. and in the 
U.K., and worked for the European Ombudsman in Brussels. She is admitted to practice in the State of New 
York, is a Solicitor in England and Wales and has also qualified as a Solicitor in the Republic of Ireland. Iria 
has a law degree (LL.B.) from Queen Mary, University of London, an LL.M. from William & Mary Law 
School, and a Ph.D. from Queen Mary, University of London. 


Ellen P. Goodman, J.D. 
Professor of Law, Co-Director and Co-Founder, Rutgers Institute for Information Policy & Law, 
Rutgers Law School (Camden) 


Ellen P. Goodman, @ellgood, is a professor of law at Rutgers Law School. She co-directs and co-founded the 
Rutgers Institute for Information Policy & Law (RIIPL). She has published widely on media and telecoms 
law, smart cities and the Internet of Things, freedom of expression, and advertising law. Goodman is currently 
a Democracy Fund grantee for a project relating to digital platform regulation and a Pratt Fund grantee for a 
project relating to algorithmic transparency. She served in the Obama Administration as a Distinguished 
Visiting Scholar with the Federal Communications Commission, and has been a visiting scholar at the London 
School of Economics and the University of Pennsylvania. She has been the recipient of Ford Foundation and 
Geraldine R. Dodge grants for work on advancing new public media models and public interest journalism. 
Prior to joining the Rutgers faculty, Goodman was a partner at the law firm of Covington & Burling LLP, 
where she practiced in the information technology area. She is a graduate of Harvard College and Harvard 
Law School, clerked for Judge Norma Shapiro on the Eastern District of Pennsylvania, and has three children. 
She also writes periodically for the Guardian and Slate on information policy. 


Kristin N. Johnson, J.D. 
McGlinchey Stafford Professor of Law and affiliate, Murphy Institute of Political Economy, Tulane 
Law School 


Professor Kristin N. Johnson is the McGlinchey Stafford Professor of Law and an affiliate of the Murphy 
Institute of Political Economy at Tulane University Law School. Professor Johnson is a leading financial 
markets regulation scholar. She has presented her research on systemic risk, risk management, cyber risk 
regulation, emerging technologies in finance, and macroprudential financial markets regulation throughout the 
United States and abroad. Professor Johnson is an elected member of the American Law Institute and an 
American Bar Association Fellow. Professor Johnson has served as a tenured Professor at Seton Hall 
University Law School and a Visiting Professor at the University of California (Irvine), the University of 
Illinois, the University of Florida, and Washington and Lee University Law Schools. Professor Johnson served 
as Assistant General Counsel and Vice President at JP Morgan and an associate at Simpson, Thatcher & 
Bartlett LLP’s New York and London offices where she represented issuers and underwriters in domestic and 
international debt and equity offerings, lenders and borrowers in banking and credit matters, and private equity 
firms and publicly-traded companies in mergers and acquisitions. Professor Johnson clerked for the Honorable 
Joseph A. Greenaway, Jr., then of the United States District Court for the District of New Jersey, elevated to 
the United States Court of Appeals for the Third Circuit. Before law school she served as an analyst at 
Goldman, Sachs & Co. Professor Johnson is a graduate of Georgetown University’s School of Foreign 


Service, cum laude, with a B.S. in Comparative Political Economy and the University of Michigan Law 
School, where she received a J.D. and served as a Notes Editor on the Michigan Law Review. 


Dhireesha Kudithipudi, Ph.D. 
Professor, Graduate Program Chair, and Director, Neuromorphic Artificial Intelligence Lab, 
Department of Computer Engineering, Rochester Institute of Technology 


Dhireesha Kudithipudi, Ph.D. is a professor and graduate program chair in the Department of Computer 
Engineering at Rochester Institute of Technology. She is the director for the Center for Human-Aware AI and 
Neuromorphic AI Lab at RIT. She holds a Ph.D. in Electrical and Computer Engineering from the University 
of Texas- San Antonio and an M.S. in Computer Engineering from Wright State University. Over the past 
decade, her research team has been paving a path to creating artificial intelligence platforms inspired by the 
brain. Dr. Kudithipudi’s team has cross-disciplinary knowledge across the neuromorphic computing stack and 
has developed neuromemristive AI platforms with continual learning capabilities. Her current research 
interests are in neuromorphic computing, brain inspired algorithms, novel computing substrates (memristors 
and 3D-ICs), energy efficient machine intelligence, and AI-on-Chip. She has secured several research grants 
in neuromorphic computing and her team has exclusive access to cutting-edge AI processors. She consults and 
collaborates with startup firms in AI. Dr. Kudithipudi is the recipient of several awards including Digital 
Rochester Technology Women of the Year, AFOSR faculty fellowship, Telluride cognitive computing 
fellowship, ASEE faculty fellowship, D&C Women to Watch, amongst others. She has authored or co- 
authored 100 manuscripts, 2 patents, and leads technical workshops in neuromorphic computing/AI. Dr. 
Kudithipudi also serves as an associate editor for IEEE Transactions on Neural Networks and Learning 
systems. 


Tom C.W. Lin, J.D. 
Professor of Law, Temple University Beasley School of Law 


Tom C.W. Lin is a Professor of Law at Temple University’s Beasley School of Law. His research and 
teaching expertise are in the areas of business organizations, corporations, securities regulation, financial 
technology, financial regulation, and compliance. Professor Lin and his research has been cited and published 
by numerous leading journals. His expertise has been featured in major media outlets like The Wall Street 
Journal, The Washington Post, Bloomberg News, and The Financial Times. His research has also been 
internationally recognized by the U.S. Securities and Exchange Commission, the U.S. Commodity Futures 
Trading Commission, and the Ontario Securities Commission, among other regulatory bodies. Professor Lin is 
a past winner of the university-wide Lindback Award for Distinguished Teaching, the George P. Williams 
Memorial Award for Outstanding Professor of the Year and the Richard J. Davis Ethics, Legal, Regulatory 
Insight Honorable Distinction Award given by the Investment Management Consultants Association. He is 
also a past nominee for the American Law Institute’s Young Scholars Medal and the Eric R. Yamamoto 
Emerging Scholar Award. He was previously on the faculty of the University of Florida Levin College of 
Law. Prior to entering academia, Professor Lin practiced law at the New York State Office of the Attorney 
General and Davis Polk & Wardwell in New York City. He holds a B.A., magna cum laude and Phi Beta 
Kappa, from New York University, and a J.D. from the University of Pennsylvania Law School, where he was 
an Arthur Littleton Fellow. 


Huu Nguyen, J.D. 
Partner, Squire Patton Boggs LLP 


Mr. Nguyen is a deal lawyer, focusing his practice on commercial and corporate transactions in the technology 
space. He counsels and assists clients with artificial intelligence arrangements, complex commercial 


arrangements, strategic relationships, financial regulatory matters, privacy and security matters, licensing, 
outsourcing, cyber law, intellectual property rights, and general technology issues. Mr. Nguyen is a vice-chair 
of the ABA’s committee on Artificial Intelligence and Robotics (2018-2019). He is also co-editor of the 
Thomson Reuters’ Fintech Law Report. Prior to being an attorney, Huu was an AI programmer, and research 
scientist. 


David W. Opderbeck, J.D., LL.M., Ph.D. 
Professor of Law, Co-Director, Gibbons Institute of Law, Science & Technology, Seton Hall University 
School of Law 


David Opderbeck is Professor of Law and Co-Director of the Gibbons Institute of Law, Science & 
Technology. His work focuses on intellectual property, cybersecurity and technology law and policy. 
Professor Opderbeck is also interested in the relationships between law, philosophy, theology, and science. He 
recently completed a Ph.D. in Systematic and Philosophical Theology at the University of Nottingham, where 
his thesis offered a philosophical and theological appraisal of the emerging discipline of “neurolaw.” 


Frank Pasquale, J.D., M.Phil. 
Professor of Law, University of Maryland Francis King Carey School of Law 


Frank Pasquale is Professor of Law at the University of Maryland’s Francis King Carey School of Law whose 
research focuses on the law of artificial intelligence, algorithms, and machine learning. He has served as a 
member of the NSF-funded Council for Big Data, Ethics, and Society, and is an Affiliate Fellow of Yale Law 
School’s Information Society Project. His book The Black Box Society: The Secret Algorithms that Control 
Money and Information (Harvard University Press, 2015) has informed the global algorithmic accountability 
movement, and has been translated into Chinese, French, and other languages. 


Oliver Round, J.D. 

Counsel & Vice President, BNY Mellon 

Oliver Round is a Counsel & Vice-President in Legal Data Management & Advisory at BNY Mellon, where 
he advises the enterprise on leveraging legal technology, including artificial intelligence, machine learning, 
contract analytics, contract generation, and contract life-cycle management. Mr. Round is counsel to the firm’s 
artificial intelligent legal agreement repository and contract analytics platform as well as to numerous 
enterprise regulatory programs. In addition, he focuses on strategically enhancing end-to-end business-as- 
usual process automation, and is a lead architect of artificial intelligence proof-of-concepts. Mr. Round is also 
regularly engaged with the trading businesses, including the roll out of the firm’s new FX Prime Brokerage 
business. Mr. Round is an alumnus of both Fordham University School of Law and Lafayette College. 


Robert Seamans, M.B.A., Ph.D. 
Associate Professor of Management and Organizations, New York University Stern School of Business 


Robert Seamans (Ph.D., UC Berkeley) is an Associate Professor at New York University’s Stern School of 
Business where he teaches courses in game theory and strategy. Professor Seamans’ research focuses on how 
firms use technology in their strategic interactions with each other. His recent work studies the economic 
consequences of AI, robotics and other advanced technologies. His research has been published in leading 
academic journals and been cited in numerous outlets including The Atlantic, Forbes, Harvard Business 
Review, New York Times, Wall Street Journal, Wired, and others. In 2015, Professor Seamans was appointed 
as the Senior Economist for technology, innovation and competition policy on President Obama’s Council of 
Economic Advisers. 


Ryan Surujnath, J.D. 
Attorney, The Blackstone Group, L.P. 


Ryan Surujnath was a Gerald Edelman Fellow for the Center for Neuroscience and Law at Fordham University 
School of Law. He is currently an in-house attorney for Blackstone. Ryan’s work with the Center focused on 
legal and policy matters relating to neuroscience-inspired artificial intelligence. His research interests also 
include finance-industry applications of blockchain technology. His work at Blackstone focuses on general 
contractual matters, fund formation, and trading system regulation. Ryan earned his J.D. from Fordham 
University School of Law in 2017. He graduated from Rutgers University in 2014, with undergraduate 
degrees in history and political science. 


Olivier Sylvain, J.D., Ph.D. 
Professor of Law, Director, McGannon Center for Communications Research, Fordham University 
School of Law 


Olivier Sylvain is a Professor of Law at Fordham University School of Law where he teaches Legislation & 
Regulation, Administrative Law, Information Law, and information law related courses. He is the Director of 
the McGannon Center for Communications Research. He is also affiliated with the Center for Law and 
Information Policy and the Center on Race, Law, and Justice. Olivier’s academic interests are chiefly in 
information and communications law and policy. He has written a variety of law review articles, symposium 
pieces, essays, policy papers, news articles, op-eds, and blog posts on current controversies in communications 
policy, online intermediary liability, privacy, and copyright. He is part of a team of legal scholars, research 
engineers, and social entrepreneurs to whom The National Science Foundation in fall 2017 awarded a three- 
year one-million-dollar grant to prototype an “edge-cloud” network that is to be owned and operated as a 
“commons resource” for Harlem residents. Before entering academia, Olivier was a Karpatkin Fellow in the 
National Legal Office of the American Civil Liberties Union in New York City and a litigation associate at 
Jenner & Block, LLC, in Washington, D.C. He is on the board of directors for the New York affiliate of the 
ACLU and teaches a class on modern American literature for local incarcerated men. 


Ari Ezra Waldman, J.D., Ph.D. 
Professor of Law, Founding Director, Innovation Center for Law and Technology, New York Law 
School 


Ari Ezra Waldman is the Professor of Law and Founding Director of the Innovation Center for Law and 
Technology at New York Law School. His research is interdisciplinary, focusing on the ways law and 
technology influence social life, privacy and the design of new technologies, and the people and structures on 
the ground that implement privacy laws on the book. In 2018, Ari published his book, Privacy as Trust: 
Information Privacy for an Information Age (Cambridge University Press), arguing that because trust is a 
primary factor in our decision to share personal information online, privacy law should be focused on 
protecting relationships of trust. Ari has been honored as the Deirdre G. Martin Memorial Lecturer on Privacy 
at the University of Ottawa and his scholarship won the Best Paper Award at the Privacy Law Scholars 
Conference in 2017 and the Otto L. Walter Distinguished Writing Award in 2016. His scholarship has been or 
will soon be published in leading law reviews and peer-reviewed academic journals, including Law and Social 
Inquiry (peer-reviewed), Washington University Law Review, Cornell Law Review Online, Notre Dame Law 
Review Online, Iowa Law Review, Indiana Law Journal, Houston Law Review, Maryland Law Review, and 
many others. He holds a Ph.D. in sociology from Columbia University, a J.D. from Harvard Law School, and 
a B.A. from Harvard College. 


Shlomit Yanisky-Ravid, Ph.D. 

Visiting Professor of Law and Head of the AI-IP Project, Center on Law and Information Policy, 
Fordham University School of Law; Lecturer in Law, Founder and Academic Director, Shalom 
Comparative Legal Research Institute, Ono Academic College, Israel 


Dr. Shlomit Yanisky-Ravid is a professor of Intellectual Property Law, focusing on IP and the challenges of 
advanced technology, artificial intelligence, blockchain, cyberspace, privacy, competition laws as well as 
comparative and international theoretical aspects of IP regime. Professor Yanisky-Ravid is a Visiting 
Professor at Fordham Law (starting 2012) where she teaches courses in IP, AI, Blockchain, and comparative 
IP. She is the Head of the “IP - AI & Blockchain Research Project” at Fordham Law CLIP. Professor 
Yanisky-Ravid is a full time Senior Law Faculty Member at the Ono Law School, in Israel. She is also a law 
professor research fellow at the Yale University Law School, ISP, since 2011 when she has done her Post- 
Doctoral studies at Yale Law School, and where she conducted two seminars. Recently, she has researched the 
challenges of AI and IP and published many articles in this field, focusing mainly on advanced technology and 
the legal regime. One of her studies, “Generating Rembrandt: Artificial Intelligence, Copyright, and 
Accountability in the 3A Era—the Human-Like Workers are Already Here—A New Model,” was chosen as 
the 2017 Visionary Article in Intellectual Property Law. She was recently identified as “the foremost thinker 
on AI and copyright” in the American Copyright Society Annual Event by Judge Katherine Forrest. Yanisky- 
Ravid is the founder and the head of the Shalom Comparative Legal Research Institute, OAC, which is active 
in Israel, Switzerland and the U.S., hosting a tremendous amount of international conferences, courses, 
seminars, visiting professors, students delegations, promoting comparative research and creating academic 
relationships with leading institutes for almost ten years. She holds B.A.s in Life Science and Psychology cum 
laude, from Bar Ilan University, Israel; an LL.B. in law, cum laude, from Tel Aviv University, Israel; a Ph.D. 
in Law, from Hebrew University, Direct Program for Outstanding Students; and a Post Doctorate from Yale 
Law School. 
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Ethically Aligned Design — Version II 
Request for Input 


Public comments are invited on the second version of Ethically Aligned Design: A Vision for 
Prioritizing Human Well-being with Autonomous and Intelligent Systems (A/IS) that encourages 
technologists to prioritize ethical considerations in the creation of such systems. 


This document has been created by committees of The IEEE Global Initiative on Ethics of 
Autonomous and Intelligent Systems, (“The IEEE Global Initiative”) composed of several hundred 
participants from six continents, who are thought leaders from academia, industry, civil society, 
policy and government in the related technical and humanistic disciplines to identify and find 
consensus on timely issues. 


The document's purpose is to: 


e Advance a public discussion about how we can establish ethical and social implementations 
for intelligent and autonomous systems and technologies, aligning them to defined values and 
ethical principles that prioritize human well-being in a given cultural context. 


e Inspire the creation of Standards (IEEE P7000™ series and beyond) and associated 
certification programs. 


e Facilitate the emergence of national and global policies that align with these principles. 


By inviting comments for Version 2 of Ethically Aligned Design, The IEEE Global Initiative provides the 
opportunity to bring together multiple voices from the related scientific and engineering communities 
with the general public to identify and find broad consensus on pressing ethical and social issues and 
candidate recommendations regarding development and implementations of these technologies. 


Input about Ethically Aligned Design should be sent by email no later than 7 May 2018 and 

will be made publicly available at the website of The /EEE Global Initiative on Ethics of Autonomous 
and Intelligent Systems no later than 4 June 2018. Details on how to submit public comments 

are available via our Submission Guidelines. 





Publicly available comments in response to this request for input will be considered by committees 
of The IEEE Global Initiative for potential inclusion in the final version of Ethically Aligned Design 
to be released in 2019. 


For further information, learn more at the website of The IEEE Global Initiative. 





If you're a journalist and would like to know more about The IEEE Global Initiative on Ethics 
of Autonomous and Intelligent Systems, please contact the IEEE-SA PR team. 
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Introduction 


As the use and impact of autonomous and intelligent systems (A/IS) become pervasive, 
we need to establish societal and policy guidelines in order for such systems to remain 
human-centric, serving humanity's values and ethical principles. These systems have 

to behave in a way that is beneficial to people beyond reaching functional goals and 
addressing technical problems. This will allow for an elevated level of trust between 
people and technology that is needed for its fruitful, pervasive use in our daily lives. 


To be able to contribute in a positive, non-dogmatic way, we, the techno-scientific 
communities, need to enhance our self-reflection, we need to have an open and 
honest debate around our imaginary, our sets of explicit or implicit values, our 
institutions, symbols and representations. 


Eudaimonia, as elucidated by Aristotle, is a practice that defines human well-being 
as the highest virtue for a society. Translated roughly as “flourishing,” the benefits 
of eudaimonia begin by conscious contemplation, where ethical considerations 
help us define how we wish to live. 


Whether our ethical practices are Western (Aristotelian, Kantian), Eastern (Shinto, 
Confucian), African (Ubuntu), or from a different tradition, by creating autonomous 

and intelligent systems that explicitly honor inalienable human rights and the beneficial 
values of their users, we can prioritize the increase of human well-being as our metric for 
progress in the algorithmic age. Measuring and honoring the potential of holistic economic 
prosperity should become more important than pursuing one-dimensional goals like 
productivity increase or GDP growth. 
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The Mission of The IEEE Global 
Initiative on Ethics of Autonomous 
and Intelligent Systems 


To ensure every stakeholder involved in the design and development of 
autonomous and intelligent systems is educated, trained, and empowered 
to prioritize ethical considerations so that these technologies are advanced 
for the benefit of humanity. 


By “stakeholder” we mean anyone involved in the research, design, manufacture, or 
messaging around intelligent and autonomous systems, including universities, organizations, 
governments, and corporations making these technologies a reality for society. 


Our goal is that Ethically Aligned Design will provide insights and recommendations 
that provide a key reference for the work of technologists in the related fields of science 
and technology in the coming years. To achieve this goal, in the current version of 
Ethically Aligned Design (EAD2v2), we identify pertinent “Issues” and “Candidate 
Recommendations” we hope will facilitate the emergence of national and global 
policies that align with these principles. 


The IEEE Global Initiative brings together several hundred participants from six continents, 
who are thought leaders from academia, industry, civil society, policy and government in the 
related technical and humanistic disciplines to identify and find consensus on timely issues. 





A second goal of The IEEE Global Initiative is to provide recommendations for IEEE 
Standards based on Ethically Aligned Design. Ethically Aligned Design (v1 and v2) and 
members of The IEEE Global Initiative are the inspiration behind the suite of IEEE P7000™ 
Standards Working Groups that are free and open for anyone to join. 
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For more information or to join any Working Group, 
please click on the links below: 


IEEE P7000™ - Model Process for Addressing Ethical Concerns During System Design 





IEEE P7001™ - Transparency of Autonomous Systems 





IEEE P7002™ - Data Privacy Process 





IEEE P7003™ - Algorithmic Bias Considerations 





IEEE P7004™ - Standard on Child and Student Data Governance 





IEEE P7005™ - Standard for Transparent Employer Data Governance 





IEEE P7006™ - Standard for Personal Data Artificial Intelligence (Al) Agent 





IEEE P7007™ - Ontological Standard for Ethically Driven Robotics and Automation Systems 





IEEE P7008™ - Standard for Ethically Driven Nudging for Robotic, Intelligent, and Automation Systems 





IEEE P7009™ - Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems 





IEEE P7010™ - Wellbeing Metrics Standard for Ethical Artificial Intelligence and Autonomous Systems 





Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Who We Are 


The IEEE Global Initiative on Ethics of 
Autonomous and Intelligent Systems (“The IEEE 
Global Initiative”) is a program of The Institute of 
Electrical and Electronics Engineers (“IEEE”), the 
world’s largest technical professional organization 
dedicated to advancing technology for the benefit 
of humanity with over 420,000 members in 
more than 160 countries. 








The IEEE Global Initiative provides the 
opportunity to bring together multiple voices 
in the related technological and scientific 
communities to identify and find consensus 
on timely issues. 








IEEE will make all versions of Ethically Aligned 
Design (EAD) available under the Creative 
Commons Attribution-Non-Commercial 3.0 
United States License. 








Subject to the terms of that license, organizations 
or individuals can adopt aspects of this work at 
their discretion at any time. It is also expected 
that EAD content and subject matter will 

be selected for submission into formal IEEE 
processes, including for standards development. 


The IEEE Global Initiative and EAD contribute to a 
broader effort at IEEE to foster open, broad, and 
inclusive conversation about ethics in technology, 
known as the IEEE TechEthics™ program. 
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Ethically Aligned Design v2 — Overview 





I. Purpose 


Intelligent and autonomous technical systems 
are specifically designed to reduce human 
intervention in our day-to-day lives. In so doing, 
these new fields are raising concerns about their 
impact on individuals and societies. Current 
discussions include advocacy for the positive 
impact, as well as warnings, based on the 
potential harm to privacy, discrimination, loss 

of skills, economic impacts, security of critical 
infrastructure, and the long-term effects on 
social well-being. Because of their nature, the full 
benefit of these technologies will be attained only 
if they are aligned with our defined values and 
ethical principles. We must therefore establish 
frameworks to guide and inform dialogue and 
debate around the non-technical implications 

of these technologies. 





Il. Goals 


The ethical design, development, and 
implementation of these technologies should be 
guided by the following General Principles: 


e Human Rights: Ensure they do not infringe 
on internationally recognized human rights 


e Well-being: Prioritize metrics of well-being 
in their design and use 


e Accountability: Ensure that their 
designers and operators are responsible 
and accountable 


e Transparency: Ensure they operate in a 
transparent manner 


e Awareness of misuse: Minimize the risks 
of their misuse 





Ill. Objectives 


Personal Data Rights and Individual 
Access Control 


A fundamental need is that people have the right 
to define access and provide informed consent 
with respect to the use of their personal digital 
data. Individuals require mechanisms to help 
curate their unique identity and personal data 

in conjunction with policies and practices that 
make them explicitly aware of consequences 
resulting from the bundling or resale of their 
personal information. 


Well-being Promoted 
by Economic Effects 


Through affordable and universal access to 
communications networks and the Internet, 
intelligent and autonomous technical systems 
can be made available to and benefit populations 
anywhere. They can significantly alter institutions 
and institutional relationships toward more 
human-centric structures and they can benefit 
humanitarian and development issues resulting 
in increased individual and societal well-being. 
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Legal Frameworks for Accountability 


The convergence of intelligent systems and 
robotics technologies has led to the development 
of systems with attributes that simulate those 

of human beings in terms of partial autonomy, 
ability to perform specific intellectual tasks, and 
may even have a human physical appearance. 
The issue of the legal status of complex intelligent 
and autonomous technical systems thus 
intertwines with broader legal questions regarding 
how to ensure accountability and allocate liability 
when such systems cause harm. Some examples 
of general frameworks to consider include the 
following: 


e — Intelligent and autonomous technical systems 
should be subject to the applicable regimes 
of property law 


* Government and industry stakeholders 
should identify the types of decisions and 
operations that should never be delegated to 
such systems and adopt rules and standards 
that ensure effective human control over 
those decisions and how to allocate legal 
responsibility for harm caused by them 


Transparency and Individual Rights 


Although self-improving algorithms and data 
analytics can enable the automation of decision- 
making impacting citizens, legal requirements 
mandate transparency, participation, and 
accuracy, including the following objectives: 


e Parties, their lawyers, and courts must have 
reasonable access to all data and information 
generated and used by such systems 
employed by governments and other 
state authorities 





* The logic and rules embedded in the system 
must be available to overseers thereof, if 
possible, and subject to risk assessments and 
rigorous testing 


e The systems should generate audit trails 
recording the facts and law supporting 
decisions and they should be amenable to 
third-party verification 


e The general public should know who is 
making or supporting ethical decisions of 
such systems through investment 


Policies for Education and Awareness 


Effective policy addresses the protection and 
promotion of safety, privacy, intellectual property 
rights, human rights, and cybersecurity, as well as 
the public understanding of the potential impact 
of intelligent and autonomous technical systems 
on society. To ensure that they best serve the 
public interest, policies should: 


e Support, promote, and enable internationally 
recognized legal norms 


e Develop workforce expertise in related 
technologies 


e — Attain research and development leadership 


e Regulate to ensure public safety and 
responsibility 


e Educate the public on societal impacts 
of related technologies 
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IV. Foundations 


Classical Ethics 


By drawing from over two thousand years’ 
worth of classical ethics traditions, The IEEE 
Global Initiative explores established ethics 
systems, addressing both scientific and religious 
approaches, including secular philosophical 
traditions, to address human morality in the 
digital age. Through reviewing the philosophical 
foundations that define autonomy and ontology, 
The IEEE Global Initiative addresses the alleged 
potential for autonomous capacity of intelligent 
technical systems, morality in amoral systems, 
and asks whether decisions made by amoral 
systems can have moral consequences. 


Well-being Metrics 


For extended intelligence and automation 
based thereupon to provably advance a specific 
benefit for humanity, there needs to be clear 
indicators of that benefit. Common metrics of 
success include profit, occupational safety, and 
fiscal health. While important, these metrics 

fail to encompass the full spectrum of well- 
being for individuals or society. Psychological, 
social, and environmental factors matter. Well- 
being metrics capture such factors, allowing the 
benefits arising from technological progress to 
be more comprehensively evaluated, providing 
opportunities to test for unintended negative 
consequences that could diminish human well- 
being. Conversely, these metrics could help 


identify where intelligent technical systems would 
increase human well-being as well, providing new 


routes to societal and technological innovation. 





Embedding Values 
into Autonomous Systems 


If machines engage in human communities as 
quasi-autonomous agents, then those agents will 
be expected to follow the community's social and 
moral norms. Embedding norms in such systems 
requires a clear delineation of the community 

in which they are to be deployed. Further, even 
within a particular community, different types of 
technical embodiments will demand different 
sets of norms. The first step is to identify the 
norms of the specific community in which the 
systems are to be deployed and, in particular, 
norms relevant to the kinds of tasks that they 
are designed to perform. 


Methodologies to Guide Ethical 
Research and Design 


To create intelligent technical systems that 
enhance and extend human well-being and 
freedom, value-based design methodologies 
put human advancement at the core of 
development of technical systems, in concert 
with the recognition that machines should serve 
humans and not the other way around. System 
developers should employ value-based design 
methodologies in order to create sustainable 
systems that can be evaluated in terms of 
both social costs and also advantages that may 
increase economic value for organizations. 
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V. Future Technology Concerns 


Reframing Autonomous Weapons 


Autonomous systems designed to cause 
physical harm have additional ethical dimensions 
as compared to both traditional weapons 

and/or autonomous systems not designed to 
cause harm. These ethical dimensions include, 
at least, the following: 


* Ensuring meaningful human control 
of weapons systems 


* Designing automated weapons with 
audit trails to help guarantee accountability 
and control 


e — Including adaptive and learning systems that 
can explain their reasoning and decisions 
to human operators in a transparent and 
understandable way 


e Training responsible human operators 
of autonomous systems who are 
clearly identifiable 


e Achieving behavior of autonomous functions 
that is predictable to their operators 


* Ensuring that the creators of these 
technologies understanding the implications 
of their work 


e Developing professional ethical codes 
to appropriately address the development 
of autonomous systems intended to 
cause harm 


Safety and Beneficence of Alleged 
Artificial General Intelligence (AGI) 
and Artificial Superintelligence (ASI) 


Similar to other powerful technologies, the 
development and use of intelligent and 





potentially self-improving technical systems 
involves considerable risk, either because of 
misuse or poor design. However, according 

to some theories, as systems approach and 
surpass AGI, unanticipated or unintended system 
behavior will become increasingly dangerous and 
difficult to correct. It is likely that not all AGI- 
level architectures can be aligned with human 
interests, and as such, care should be taken 

to determine how different architectures will 
perform as they become more capable. 


Affective Computing 


Affect is a core aspect of intelligence. Drives and 
emotions such as anger, fear, and joy are often 
the foundations of actions throughout our life. 
To ensure that intelligent technical systems will 
be used to help humanity to the greatest extent 
possible in all contexts, artifacts participating in 
or facilitating human society should not cause 
harm either by amplifying or damping human 
emotional experience. Even the rudimentary 
versions of synthetic emotions already deployed 
in some systems impact how they are perceived 
by policy makers and the general public. 


Mixed Reality 


Mixed reality could alter our concepts of identity 
and reality as these technologies become more 
common in our work, education, social lives, and 
commercial transactions. The ability for real-time 
personalization of this mixed-reality world raises 
ethical questions concerning the rights of the 
individual and control over one’s multifaceted 
identity, especially as the technology moves 
from headsets to more subtle and integrated 
sensory enhancements. 
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Our Process 


To ensure greatest cultural relevance and 
intellectual rigor in our work, The IEEE Global 
Initiative has been globally crowdsourcing 
feedback for Versions 1 and 2 of Ethically 
Aligned Design. 


We released Ethically Aligned Design Version 1 
as a Request for Input on December of 2016 

and received over two hundred pages of in-depth 
feedback about the draft. As a way to highlight 
insights inspired by the feedback we received, 
Sara Mattingly-Jordan of The IEEE Global Initiative 
also wrote the report, Becoming a Leader in 
Global Ethics. 











We are releasing Ethically Aligned Design Version 
2 (EADv2) as a Request for Input once again 

to gain further insights about the eight original 
sections from EADv1, along with unique/new 
feedback for the five new sections included 

in EADv2. 





Next Steps 


The IEEE Global Initiative is currently creating 
an organizational committee composed of 
representatives of all our Committees and IEEE 
P7000™ Working Groups to do the following 
in order to prepare the final version of Ethically 
Aligned Design to be released in 2019: 


e Create criteria for Committees to vote on all 
“Candidate Recommendations” becoming 
“Recommendations” based on the General 
Principles of Ethically Aligned Design that are 


creative 
commons 


in accordance with the Mission Statement of 
The IEEE Global Initiative. This voting process 
will be based on the consensus-based 
protocols provided by IEEE-SA. 


e Create a rigorous methodology to best 
incorporate feedback received from EADv1 
and EADv2, working to holistically consider 
global and diversity-based considerations 
for content inclusion. 


e Use the glossary we have produced as a key 
tool for synthesizing content for final version 
of EAD, unifying terms as much as possible. 


Final Version of Ethically Aligned 
Design — Format and Goals 


The final version of Ethically Aligned Design will 
be made available in the following formats: 


e Handbook. While specific formatting is 
still under consideration, the final version 
of Ethically Aligned Design will feature 
“Recommendations” (versus “Candidate 
Recommendations”) for all existing and 
future “Issues” voted on by Members of 
The IEEE Global Initiative. It is very likely 
the final version of EAD will not be broken 
into sections according to Committees (as 
with EADv1 and EADv2) but according to 
themes or principles to be decided on by 
the organizational committee mentioned 
above. While not an official IEEE position 
statement, “Recommendations” will be 
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created to be easily utilized by technologists 
and policy makers focusing on autonomous 
and intelligent systems design, usage, and 
governance. 


e Educational materials. The IEEE Global 
Initiative would like to convert the handbook 
version of Ethically Aligned Design into an 
academically oriented book/educational 
materials. Evergreen in nature, these would 
be targeted to academics, engineers, and 
technologists looking for global guidance 
to be used in university, post-grad, or 
other educational settings where ethics in 
technology or the issues EAD comprises 
would be taught. 


Incorporating Feedback 


While it was our intention to directly accept or 
review all feedback we received for EADv1, we 
were (happily) overwhelmed with the fantastic 
response we received. However, to most 
holistically include feedback from EADv1 and 
EADv2 into our overall process we have created 
a Glossary and are working to increase more 
global representation and diversity in our work. 
Specifically: 


Glossary 


We received a great deal of feedback on the 
need for aligned recommendations for key 
terms in Ethically Aligned Design. To that 
end, we created a Glossary Committee and 
launched the first draft of our Glossary at the 
same time we released EADv2. Our goal is 
to refine our Glossary so that by mid-2018, 


based on aggregated feedback to all sections 

of EAD (Versions 1 and 2), we can standardize 
definitions that reflect a global and holistic set of 
definitions to be implemented by all Committees 
in the final version of EAD. 


More Global Representation/Diversity 


We received a great deal of feedback noting 
that EADv1 was fairly “Western” in its cultural 
orientation. This makes sense, as the initial 100 
members working on EADv1 where largely from 
North America and the European Union. Since 
the release of EADv1, however, we have: 


e Added members from China, Korea, Japan, 
Brazil, Mexico, the Russian Federation, Iran, 
Thailand, and Israel along with new people 
from the United States and the European 
Union. In addition to the 250 members of 
the Initiative, there are also now more than 
400 global members in the IEEE P7000™ 
Working Groups that EAD inspired. 


* Supported the members translating 
the Executive Summary of EADv1 into 
multiple languages. 


e Added our new “Classical Ethics in A/IS” 
Committee. 


* Created the Becoming a Leader in Global 
Ethics report. 





e Commissioned a report from our newer 
global members about the state of A/IS 
Ethics in their regions. 








e Created an Outreach Committee to help 
identify and incorporate work being done 
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in A/IS ethics by women, people of color, 
students, and other groups representing the 
full spectrum of society that we are hoping 
to positively influence with our work. We 
are currently working with members of The 
Reboot Retreat, AI4ALL, and other leaders 
within IEEE to help us ensure that The IEEE 
Global Initiative and the final version of 
Ethically Aligned Design are as holistically 
representative and relevant as possible. 





Terminology Update 


There is no need to use the term artificial 
intelligence in order to conceptualize and speak 
of technologies and systems that are meant to 
extend our human intelligence or be used in 
robotics applications. For this reason, we use 
the term, autonomous and intelligent systems 
(or A/IS) in the course of our work. We chose 
to use this phrase encapsulating multiple 

fields (machine learning, intelligent systems 


engineering, robotics, etc.) throughout Ethically 
Aligned Design, Version 2 to ensure the broadest 
application of ethical considerations in 

the design of these technologies as possible. 


How the Document Was Prepared 


This document was prepared using an open, 
collaborative, and consensus building approach, 
following the processes of the Industry 
Connections program, a program of the 

IEEE Standards Association. 





Industry Connections facilitates collaboration 
among organizations and individuals as they 
hone and refine their thinking on emerging 
technology issues, helping to incubate potential 
new standards activities and standards-related 
products and services. 
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How to Cite Ethically Aligned Design 


Please cite Version 2 of Ethically Aligned Design Aligned Design: A Vision for Prioritizing Human 
in the following manner: Well-being with Autonomous and Intelligent 
Systems, Version 2. IEEE, 2017. http://standards. 
ieee.org/develop/indconn/ec/autonomous_ 
systems.html. 


The IEEE Global Initiative on Ethics of 
Autonomous and Intelligent Systems. Ethically 
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and Mark Halverson 
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The Future of Life Institute's Asilomar Al 
Principles, The Al Now 2017 Report, Human 
Rights in the Robot Age Report from The 
Rathenau Instituut, Report of COMEST on 
Robotics Ethics from UNESCO, The European 
Parliament's Recommendations to the 
Commission on Civil Law Rules on Robotics, 
Artificial intelligence — The Consequences of 
Artificial Intelligence on the (Digital) Single 
Market, Production, Consumption, Employment 
and Society report from the European Economic 
and Social Committee (Rapporteur: Catelijne 
MULLER), OECD's report, Going Digital: Making 
the Transformation Work for Growth and Well- 
Being, USACM's Statement on Algorithmic 
Transparency and Accountability, Guide to the 
Ethical Design and Application of Robots and 
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la Recherche en Robotique (CERNA), Charta 

der Digitalen Grundrechte der Europäischen 
Union (Charter of the Digital Fundamental Rights 
of the European Union), Telecommunications 
Research Laboratory, “Al Network Kent kai 

Kaigi H kokusho 2016: Al Network no Eiky to 
Risk — Chiren Shakai (WINS) no Jitsugen ni 
Muketa Kadai” (AIZY PI— JALIR RTA RR HR 
#82016 AR- MRIKI- INEI 
AZ -EEA (WINS(I1 YVA)) ORRA 
(RRE - 1) [The Conference on Networking 
among Als Report (2016): Impacts and Risks 

of Al Networking Issues for the Realization of 
Wisdom Network Society, (WINS)], Japanese 
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The Information Technology Industry Council's 
Al Policy Principles, Intel’s Artificial Intelligence 

— The Public Policy Opportunity, IEEE European 
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Artificial Intelligence: Calling on Policy Makers 

to Take a Leading Role in Setting a Long Term 

Al Strategy, IEEE-USA’s position statement on 
Artificial Intelligence Research, Development and 
Regulation, The IEEE Global Initiative’s Prioritizing 
Human Well-being in the Age of Artificial 


Intelligence. 
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Organizations 


The Association for the Advancement of Artificial 
Intelligence and their formative work on Al Ethics, 
The Future of Life Institute, The Partnership on 

Al to Benefit People and Society, The Foundation 














for Responsible Robotics, Al & Society, Machine 
Intelligence Research Institute, The International 
Center for Information Ethics, The African Center 
of Excellence for Information Ethics, The 4TU 
Center for Ethics and Technology, The Center 
for the Study of Existential Risk, The Leverhulme 
Center for the Future of Intelligence, The Future 
of Humanity Institute, The Japanese Society 

for Artificial Intelligence, The Association for 
Computing Machinery, Future Advocacy, ACM 
Special Interest Group on Artificial Intelligence, 
The World Economic Forum's Global Future 
Council of Artificial Intelligence and Robotics, 
The Digital Asia Hub, The Al Initiative, The Open 
Roboethics Institute, The Dalai Lama Center for 
Ethics and Transformative Values at MIT, The 
Ethics Initiative at MIT Media Lab, The IEEE- 
USA Government Relations Council Artificial 
Intelligence Committee, The IEEE Robotics 

and Automation Society Committee on Robot 
Ethics, The IEEE Robotics and Automation 
Society, The IEEE Society on Social Implications 
of Technology, The IEEE Computer Society, The 
IEEE Computational Intelligence Society, The IEEE 
















































































Systems, Man and Cybernetics Society, The IEEE 
Symbiotic Autonomous Systems Initiative. 
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and constant support of The IEEE Global Initiative 
by Dr. Ing. Konstantinos Karachalios, Managing 
Director of the IEEE Standards Association and a 
member of the IEEE Management Council. 


We would especially like to thank Eileen M. Lach, 
the IEEE General Counsel and Chief Compliance 
Officer, who invested her time and expertise in 
fully reviewing this entire document, with the 
heartfelt conviction that there is a pressing need 
to focus the global community on highlighting 
ethical considerations in the development of 
autonomous and intelligent systems. 


Special thanks to Dr. Peter S. Brooks for his 
contributions to the Overview of EADv2. 


Thank You to Our Members 
and IEEE Team 


Our progress and the ongoing positive influence 
of this work is due to the volunteer experts 
serving on our Committees and IEEE P7000™ 
Standards Working Groups, along with the IEEE 
staff who support our efforts. Thank you for 

your dedication toward defining, designing, and 
inspiring the ethical PRINCIPLES and STANDARDS 
that will ensure that intelligent and autonomous 
systems and the technologies associated 
therewith will positively benefit humanity. 
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Disclaimers 


Ethically Aligned Design is not a code of conduct 
or a professional code of ethics. Engineers and 
technologists have well-established codes, 

and we wish to respectfully recognize the 
formative precedents surrounding issues of 
ethics and safety and the professional values 
these codes represent. These codes provide the 
broad framework for the more focused domain 
addressed in this document, and it is our hope 
that the inclusive, consensus-building process 
around its design will contribute unique value to 
technologists and society as a whole. 


This document is also not a position, or policy 
statement, or formal report of IEEE or any 

other organization with which is affiliated. It is 
intended to be a working reference tool created 
in an inclusive process by those in the relevant 
scientific and engineering communities prioritizing 
ethical considerations in their work. 


A Note on Affiliations Regarding 
Members of The Initiative 


The language and views expressed in Ethically 
Aligned Design reflect the individuals who 
created content for each section of this 
document. The language and views expressed 
in this document do not necessarily reflect 

the positions taken by the universities or 
organizations to which these individuals belong, 
and should in no way be considered any form 
of endorsement, implied or otherwise, from 
these institutions. 
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This is the second version of Ethically Aligned 
Design. Where individuals are listed in a 
Committee it indicates only that they are 
Members of that Committee. Committee 
Members may not have achieved final 
concurrence on content in this document 
because of its versioning format and the 
concurrence-building process of The IEEE Global 
Initiative. Content listed by Members in this 

or future versions is not an endorsement, implied 
or otherwise, until formally stated as such. 


A Note Regarding Candidate 
Recommendations in This Document 


Ethically Aligned Design is being created 

via multiple versions that are being iterated 
over the course of two to three years. The 
IEEE Global Initiative is following a specific 
concurrence-building process where members 
contributing content are proposing candidate 
recommendations so as not to imply these are 
final recommendations at this time. 


Our Membership 


The IEEE Global Initiative currently has more than 
250 experts from all but one continent involved 
in our work, and we are eager for new voices and 
perspectives to join our work. 
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Notice and Disclaimer of Liability 
Concerning the Use of IEEE-SA Industry 
Connections Documents 


This IEEE Standards Association (“IEEE-SA") 
Industry Connections publication (“Work”) is not 
a consensus standard document. Specifically, this 
document is NOT AN IEEE STANDARD. Information 
contained in this Work has been created by, or 
obtained from, sources believed to be reliable, 
and reviewed by members of the IEEE-SA Industry 
Connections activity that produced this Work. 

IEEE and the IEEE-SA Industry Connections 
activity members expressly disclaim all warranties 
(express, implied, and statutory) related to this 
Work, including, but not limited to, the warranties 
of: merchantability; fitness for a particular purpose; 
non-infringement; quality, accuracy, effectiveness, 
currency, or completeness of the Work or content 
within the Work. In addition, IEEE and the IEEE- 
SA Industry Connections activity members 
disclaim any and all conditions relating to: results; 
and workmanlike effort. This IEEE-SA Industry 
Connections document is supplied “AS IS” 

and “WITH ALL FAULTS.” 


Although the IEEE-SA Industry Connections 
activity members who have created this Work 
believe that the information and guidance 
given in this Work serve as an enhancement 
to users, all persons must rely upon their own 
skill and judgment when making use of it. IN 
NO EVENT SHALL IEEE OR IEEE-SA INDUSTRY 
CONNECTIONS ACTIVITY MEMBERS BE LIABLE 
FOR ANY ERRORS OR OMISSIONS OR DIRECT, 
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, 
OR CONSEQUENTIAL DAMAGES (INCLUDING, 
BUT NOT LIMITED TO: PROCUREMENT OF 
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SUBSTITUTE GOODS OR SERVICES; LOSS 
OF USE, DATA, OR PROFITS; OR BUSINESS 
INTERRUPTION) HOWEVER CAUSED AND 
ON ANY THEORY OF LIABILITY, WHETHER 
IN CONTRACT, STRICT LIABILITY, OR TORT 
(INCLUDING NEGLIGENCE OR OTHERWISE) 
ARISING IN ANY WAY OUT OF THE USE OF 
THIS WORK, EVEN IF ADVISED OF THE 
POSSIBILITY OF SUCH DAMAGE AND 
REGARDLESS OF WHETHER SUCH 
DAMAGE WAS FORESEEABLE. 


Further, information contained in this Work 
may be protected by intellectual property rights 
held by third parties or organizations, and the 
use of this information may require the user to 
negotiate with any such rights holders in order 
to legally acquire the rights to do so, and such 
rights holders may refuse to grant such rights. 
Attention is also called to the possibility that 
implementation of any or all of this Work may 
require use of subject matter covered by patent 
rights. By publication of this Work, no position 
is taken by IEEE with respect to the existence 
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or validity of any patent rights in connection 
therewith. IEEE is not responsible for identifying 
patent rights for which a license may be required, 
or for conducting inquiries into the legal validity 
or scope of patents claims. Users are expressly 
advised that determination of the validity of any 
patent rights, and the risk of infringement of 

such rights, is entirely their own responsibility. 

No commitment to grant licenses under patent 
rights on a reasonable or non-discriminatory basis 
has been sought or received from any rights 
holder. The policies and procedures under which 
this document was created can be viewed at 
standards.ieee.org/about/sasb/iccom/. 





This Work is published with the understanding 
that IEEE and the IEEE-SA Industry Connections 
activity members are supplying information 
through this Work, not attempting to render 
engineering or other professional services. 

If such services are required, the assistance 

of an appropriate professional should be sought. 
IEEE is not responsible for the statements and 
opinions advanced in this Work. 
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The General Principles Committee seeks to articulate high-level ethical concerns that 

apply to all types of autonomous and intelligent systems (A/IS*), regardless of whether 
they are physical robots (such as care robots or driverless cars) or software systems 

(such as medical diagnosis systems, intelligent personal assistants, or algorithmic chat bots). 
We are motivated by a desire to create ethical principles for A/IS that: 


1. Embody the highest ideals of human beneficence as a superset of Human Rights. 


2. Prioritize benefits to humanity and the natural environment from the use of A/IS. 
Note that these should not be at odds — one depends on the other. Prioritizing human 
well-being does not mean degrading the environment. 


3. Mitigate risks and negative impacts, including misuse, as A/IS evolve as socio-technical 
systems. In particular by ensuring A/IS are accountable and transparent. 


It is our intention that by identifying issues and drafting recommendations these principles 
will serve to underpin and scaffold future norms and standards within a framework of 
ethical governance. 


We have identified principles created by our Committee as well as aggregated principles 
reflected from other Committees of The IEEE Global Initiative. Therefore, readers should 
note that some general principles are reiterated and elaborated by other committees, as 
appropriate to the specific concerns of those committees. We have purposefully structured 
our Committee and this document in this way to provide readers with a broad sense 

of the themes and ideals reflecting the nature of ethical alignment for these technologies 
as an introduction to our overall mission and work. 
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The following provides high-level guiding principles for potential solutions-by-design 
whereas other Committee sections address more granular issues regarding specific 
contextual, cultural, and pragmatic questions of their implementation. 

*The acronym A/IS is shorthand for Autonomous and Intelligent Systems. When represented in this way, it refers to the 
overlapping concerns about the design, development, deployment, decommissioning, and adoption of autonomous or intelligent 


software when installed into other software and/or hardware systems that are able to exercise independent reasoning, 
decision-making, intention forming, and motivating skills according to self-defined principles. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Principle 1 — Human Rights 





Issue: 


How can we ensure that 
A/IS do not infringe upon 
human rights? 


Background 


Human benefit is an important goal of A/IS, 

as is respect for human rights set out, inter alia, 
in The Universal Declaration of Human Rights, 
the International Covenant for Civil and Political 
Rights, the Convention on the Rights of the 
Child, Convention on the Elimination of all forms 
of Discrimination against Women, Convention 
on the Rights of Persons with Disabilities, and 
the Geneva Conventions. Such rights need to 
be fully taken into consideration by individuals, 
companies, professional bodies, research 
institutions, and governments alike to reflect 
the following concerns: 




















1. A/IS should be designed and operated in 
a way that both respects and fulfills human 
rights, freedoms, human dignity, and cultural 
diversity. 


2. A/IS must be verifiably safe and secure 
throughout their operational lifetime. 
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3. Ifan A/IS causes harm it must always 
be possible to discover the root cause, 
by assuring traceability for said harm 
(see also Principle 4 — Transparency). 


While their interpretation may change over time, 
human rights as defined by international law, 
provide a unilateral basis of creating any A/IS 
system as they affect humans, their emotions, 
data, or agency. While the direct coding of human 
rights in A/IS may be difficult or impossible based 
on contextual use, newer guidelines from The 
United Nations, such as the Ruggie principles, 
provide methods to pragmatically implement 
human rights ideals within business or corporate 
contexts that could be adapted for engineers and 
technologists. In this way technologists can take 
account of rights in the way A/IS are operated, 
tested, validated, etc. In short, human rights 
should be part of the ethical risk assessment 

of A/IS. 





Candidate Recommendations 


To best honor human rights, society must assure 
the safety and security of A/IS so that they are 
designed and operated in a way that benefits 
humans: 


1. Governance frameworks, including standards 
and regulatory bodies, should be established 
to oversee processes assuring that the 
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use of A/IS does not infringe upon human 
rights, freedoms, dignity, and privacy, and of 
traceability to contribute to the building 

of public trust in A/IS. 


2. Away to translate existing and forthcoming 
legal obligations into informed policy and 
technical considerations is needed. Such 
a method should allow for differing cultural 
norms as well as legal and regulatory 
frameworks. 


3. For the foreseeable future, A/IS should 
not be granted rights and privileges equal 
to human rights: A/IS should always be 
subordinate to human judgment and control. 


Further Resources 


The following documents/organizations are 
provided both as references and examples of 
the types of work that can be emulated, adapted, 
and proliferated, regarding ethical best practices 
around A/IS to best honor human rights: 


* The Universal Declaration of Human Rights, 
1947. 





e The International Covenant on Civil and 
Political Rights, 1966. 
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The International Covenant on Economic, 
Social and Cultural Rights, 1966. 








The International Convention on 
the Elimination of All Forms of Racial 
Discrimination, 1965. 








The Convention on the Rights of the Child. 





The Convention on the Elimination of All 
Forms of Discrimination against Women, 
1979. 








The Convention on the Rights of Persons 
with Disabilities, 2006. 





The Geneva Conventions and additional 
protocols, 1949. 


IRTF’s Research into Human Rights Protocol 
Considerations. 





The UN Guiding Principles on Business 
and Human Rights, 2011. 








For an example of a guide on how to conduct 
an ethical risk assessment see British Standards 
Institute BS8611:2016, Guide to the Ethical 
Design and Application of Robots and 


Robotic Systems. 
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Principle 2 — Prioritizing Well-being 





Issue: 


Traditional metrics of prosperity 
do not take into account the 
full effect of A/IS technologies 
on human well-being. 


Background 


A focus on creating ethical and responsible 
Al has been increasing among technologists 
in the past 12 to 16 months. Key issues of 
transparency, accountability, and algorithmic 
bias are being directly addressed for the 
design and implementation of A/IS. While 
this is an encouraging trend, a key question 
facing technologists today is beyond designing 
responsible A/IS. That question is, What are 
the specific metrics of societal success for 
“ethical Al” once released to the world? 


For A/IS technologies to provably advance 

benefit for humanity, we need to be able 

to define and measure the benefit we wish 

to increase. Avoiding negative unintended 
consequences and increasing value for customers 
and society (today measured largely by gross 
domestic product (GDP), profit, or consumption 
levels) are often the only indicators utilized 

in determining success for A/IS. 
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Well-being, for the purpose of The IEEE Global 
Initiative, is defined as encompassing human 
satisfaction with life and the conditions of life as 
well as an appropriate balance between positive 
and negative affect. This definition is based on 
the Organization for Economic Co-Operation and 
Development's (OECD) Guidelines on Measuring 
Subjective Well-being that notes, “Being able to 
measure people's quality of life is fundamental 
when assessing the progress of societies. There 
is now widespread acknowledgement that 
measuring subjective well-being is an essential 
part of measuring quality of life alongside other 
social and economic dimensions.” Data is 

also currently being gathered in governments, 
businesses, and other institutions using 
scientifically valid measurements of well-being. 
Since modern societies are largely constituted 

of A/IS users, we believe these considerations 
to be relevant for A/IS developers. 








It is widely agreed that GDP is at best incomplete, 
and at worst misleading, as a metric of 

true prosperity for society at large and A/IS 
technologies (as noted in The Oxford Handbook 
of Well-Being and Public Policy). Although the 
concerns regarding GDP reflect holistic aspects 
of society versus the impact of any one 
technology, they reflect the lack of universal 
usage of well-being indicators for A/IS. A/IS 
undoubtedly hold positive promise for society. 
But beyond the critical importance of designing 
and manufacturing these technologies in an 
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ethically driven and responsible manner is 
the seminal question of determining the key 
performance indicators (KPIs) of their success 
once introduced into society. 


A/IS technologies can be narrowly conceived 
from an ethical standpoint; be legal, profitable, 
and safe in their usage; and yet not positively 
contribute to human well-being. This means 
technologies created with the best intentions, 
but without considering well-being metrics, 
can still have dramatic negative consequences 
on people's mental health, emotions, sense 
of themselves, their autonomy, their ability 

to achieve their goals, and other dimensions 
of well-being. 


Nonetheless, quantitative indicators of individual 
well-being should be introduced with caution, 
as they may provoke in users an automatic urge 
for numerical optimization. While this tendency 
is theoretically unavoidable, efforts should be 
invested in guaranteeing that it will not flatten the 
diversity of human experience. The A/IS using 
quantitative indicators for health or happiness 
should therefore develop and implement 
measures for maintaining full human autonomy 
of their users. 


In conclusion, it is widely agreed that de facto 
metrics regarding safety and fiscal health do not 
encompass the full spectrum of well-being for 
individuals or society. By not elevating additional 
environmental and societal indicators as pillars 

of success for A/IS, we risk minimizing the 
positive and holistic impact for humanity of these 
technologies. Where personal, environmental, 

or social factors are not prioritized as highly as 


fiscal metrics of success, we also risk expediting 
negative and irreversible harms to our planet 
and population. 


Candidate Recommendation 


A/IS should prioritize human well-being as an 
outcome in all system designs, using the best 
available, and widely accepted, well-being metrics 
as their reference point. 


Further Resources 


* IEEE P7010™, Well-being Metrics Standard 
for Ethical Al and Autonomous Systems. 








* The Measurement of Economic Performance 
and Social Progress (2009) now commonly 
referred to as “The Stiglitz Report,” 
commissioned by the then President of 
the French Republic. From the report: “... 
the time is ripe for our measurement system 
to shift emphasis from measuring economic 
production to measuring people's well-being 
... emphasizing well-being is important 
because there appears to be an increasing 
gap between the information contained 
in aggregate GDP data and what counts 
for common people's well-being.” 








e Organisation for Economic Co-Operation 
& Development, OECD Guidelines for 
Measuring Subjective Well-being. Paris: 
OECD, 2013. 


* Beyond GDP (European Commission) 
From the site: “The Beyond GDP initiative 
is about developing indicators that are 
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as clear and appealing as GDP, but more 
inclusive of environmental and social aspects 
of progress.” 


Global Dialogue for Happiness, part of 
the annual World Government Summit, 
February 11, 2017. 





Organization for Economic Co-Operation 
and Development, OECD's Better Life Index. 





New Economics Foundation, The Happy 
Planet Index. 


Redefining Progress, Genuine Progress 
Indicator. 





The International Panel on Social Progress, 
Social Justice, Well-Being and Economic 


Organization. 





Veenhoven, R. World Database of Happiness. 
Rotterdam, The Netherlands: Erasmus 
University. 





Royal Government of Bhutan. The Report 

of the High-Level Meeting on Wellbeing 

and Happiness: Defining a New Economic 
Paradigm. New York: The Permanent Mission 
of the Kingdom of Bhutan to the United 
Nations, 2012. 








See also Well-being Section in Ethically 
Aligned Design, Version 2. 
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Principle 3 — Accountability 





Issue: 


How can we assure that 
designers, manufacturers, 
owners, and operators of A/IS 
are responsible and accountable? 


Background 


The programming, output, and purpose of A/IS 
are often not discernible by the general public. 
Based on the cultural context, application, 

and use of A/IS, people and institutions need 
clarity around the manufacture and deployment 
of these systems to establish responsibility 

and accountability, and avoid potential harm. 
Additionally, manufacturers of these systems 
must be able to provide programmatic-level 
accountability proving why a system operates in 
certain ways to address legal issues of culpability, 
if necessary apportion culpability among several 
responsible designers, manufacturers, owners, 
and/or operators, to avoid confusion or fear 
within the general public. 


Note that accountability is enhanced with 
transparency, thus this principle is closely linked 
with Principle 4 — Transparency. 
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Candidate Recommendations 


To best address issues of responsibility and 
accountability: 


k 


Legislatures/courts should clarify issues 

of responsibility, culpability, liability, and 
accountability for A/IS where possible during 
development and deployment (so that 
manufacturers and users understand their 
rights and obligations). 


Designers and developers of A/IS should 
remain aware of, and take into account when 
relevant, the diversity of existing cultural 
norms among the groups of users of these 
A/IS. 


Multi-stakeholder ecosystems should be 
developed to help create norms (which can 
mature to best practices and laws) where 
they do not exist because A/IS-oriented 
technology and their impacts are too new 
(including representatives of civil society, 
law enforcement, insurers, manufacturers, 
engineers, lawyers, etc.). 


Systems for registration and record-keeping 
should be created so that it is always possible 
to find out who is legally responsible for 

a particular A/IS. Manufacturers/operators/ 
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owners of A/IS should register key, high-level 


parameters, including: 
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Intended use 


Training data/training environment 
(if applicable) 


Sensors/real world data sources 
Algorithms 

Process graphs 

Model features (at various levels) 
User interfaces 
Actuators/outputs 


Optimization goal/loss function/reward 
function 
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Further Resources 


Shneiderman, B. “Human Responsibility 
for Autonomous Agents.” /EEE Intelligent 
Systems 22, no. 2, (2007): 60-61. 


Matthias, A. “The Responsibility Gap: 
Ascribing Responsibility for the Actions of 
Learning Automata.” Ethics and Information 
Technology 6, no. 3 (2004): 175-183. 


Hevelke A., and J. Nida-Riimelin. 
“Responsibility for Crashes of Autonomous 
Vehicles: An Ethical Analysis.” Science 

and Engineering Ethics 21, no. 3 (2015): 
619-630. 


An example of good practice (in relation 

to Candidate Recommendation #3) can 

be found in Sciencewise — the U.K. national 
center for public dialogue in policy-making 
involving science and technology issues. 
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Principle 4 — Transparency 





Issue: 


How can we ensure that A/IS 
are transparent? 


Background 


A key concern over autonomous systems is 
that their operation must be transparent to a 
wide range of stakeholders for different reasons 
(noting that the level of transparency will 
necessarily be different for each stakeholder). 
Stated simply, transparent A/IS are ones in which 
it is possible to discover how and why a system 
made a particular decision, or in the case of a 
robot, acted the way it did. Note that here the 
term transparency also addresses the concepts 
of traceability, explicability, and interpretability. 


A/IS will be performing tasks that are far more 
complex and have more effect on our world 
than prior generations of technology. This reality 
will be particularly acute with systems that 
interact with the physical world, thus raising the 
potential level of harm that such a system could 
cause. For example, some A/IS already have 
real consequences to human safety or well- 
being, such as medical diagnosis Al systems, or 
driverless car autopilots; systems such as these 
are safety-critical systems. 


At the same time, the complexity of A/IS 
technology will make it difficult for users of 
those systems to understand the capabilities 
and limitations of the Al systems that they 

use, or with which they interact. This opacity, 
combined with the often-decentralized manner 
in which it is developed, will complicate efforts 
to determine and allocate responsibility when 
something goes wrong with an Al system. Thus, 
lack of transparency both increases the risk and 
magnitude of harm (users not understanding the 
systems they are using) and also increases the 
difficulty of ensuring accountability (see Principle 
3— Accountability). 


Transparency is important to each stakeholder 
group for the following reasons: 


1. For users, transparency is important 
because it provides a simple way for them 
to understand what the system is doing 
and why. 


2. For validation and certification of an A/IS, 
transparency is important because it exposes 
the system's processes and input data 
to scrutiny. 


3. If accidents occur, the AS will need to be 
transparent to an accident investigator, so the 
internal process that led to the accident can 
be understood. 
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4. Following an accident, judges, juries, lawyers, 
and expert witnesses involved in the trial 
process require transparency to inform 
evidence and decision-making. 


5. For disruptive technologies, such as driverless 
cars, a certain level of transparency to wider 
society is needed to build public confidence 
in the technology, promote safer practices, 
and facilitate wider societal adoption. 


Candidate Recommendation 


Develop new standards* that describe measurable, 
testable levels of transparency, so that systems 
can be objectively assessed and levels of 
compliance determined. For designers, such 
standards will provide a guide for self-assessing 
transparency during development and suggest 
mechanisms for improving transparency. (The 
mechanisms by which transparency is provided 
will vary significantly, for instance 1) for users 

of care or domestic robots, a why-did-you-do-that 
button which, when pressed, causes the robot 

to explain the action it just took, 2) for validation 
or certification agencies, the algorithms underlying 
the A/IS and how they have been verified, and 

3) for accident investigators, secure storage 

of sensor and internal state data, comparable 

to a flight data recorder or black box.) 


*Note that IEEE Standards Working Group 
P7001™ has been set up in response to this 
recommendation. 
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Principle 5 — A/IS Technology Misuse 


and Awareness of It 





Issue: 


How can we extend the benefits 
and minimize the risks of A/IS 
technology being misused? 


Background 


New technologies give rise to greater risk of 
misuse, and this is especially true for A/IS. A/IS 
increases the impact of risks such as hacking, the 
misuse of personal data, “gaming,” or exploitation 
(e.g., of vulnerable users by unscrupulous 
parties). These are not theoretical risks. Cases of 
A/IS hacking have already been widely reported, 
of driverless cars for example. The EU’s General 
Data Protection Regulation (GDPR) provides 
measures to remedy the misuse of personal 
data. The Microsoft Tay Al chatbot was famously 
gamed when it mimicked deliberately offensive 
users. In an age where these powerful tools are 
easily available, there is a need for new kind of 
education for citizens to be sensitized to risks 
associated with the misuse of A/IS. 





Responsible innovation requires designers to 
anticipate, reflect, and engage with users of A/IS 
thus, through education and awareness, citizens, 
lawyers, governments, etc. have a role to play in 
developing accountability structures (Principle 3). 
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They also have a role to play in guiding new 
technology proactively toward beneficial ends. 


Candidate Recommendations 


Raise public awareness around the issues of 
potential A/IS technology misuse in an informed 
and measured way by: 


1. Providing ethics education and security 
awareness that sensitizes society to the 
potential risks of misuse of A/IS (e.g., 
by providing “data privacy” warnings that 
some smart devices will collect their user's 
personal data). 


2. Delivering this education in scalable and 
effective ways, beginning with those having 
the greatest credibility and impact that also 
minimize generalized (e.g., non-productive) 
fear about A/IS (e.g., via credible research 
institutions or think tanks via social media 
such as Facebook or YouTube). 


3. Educating government, lawmakers, and 
enforcement agencies surrounding these 
issues so citizens work collaboratively with 
them to avoid fear or confusion (e.g., in the 
same way police officers have given public 
safety lectures in schools for years; in the 
near future they could provide workshops 
on safe A/IS). 
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Further Resources + Engineering and Physical Sciences Research 
Council. Anticipate, Reflect, Engage and Act 
(AREA) Framework for Responsible Research 
and Innovation. 


e Greenberg, A. “Hackers Fool Tesla S's 
Autopilot to Hide and Spoof Obstacles.” 
Wired, August 2016. 











e (In relation to Candidate Recommendation 
#2) Wilkinson, C., and E. Weitkamp. 
Creative Research Communication: Theory 





and Practice. Manchester, UK: Manchester 
University Press, 2016. 
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Society has not established universal standards or guidelines for embedding human norms 
and values into autonomous and intelligent systems (A/IS) today. But as these systems are 
instilled with increasing autonomy in making decisions and manipulating their environment, 
it is essential they be designed to adopt, learn, and follow the norms and values of the 
community they serve. Moreover, their actions must be transparent in signaling their norm 
compliance and, if needed, they must be able to explain their actions. This is essential 

if humans are to develop levels of trust in A/IS that are appropriate in the specific contexts 
and roles in which A/IS function. 


The conceptual complexities surrounding what “values” are (e.g., Hitlin and Piliavin, 2004; 
Malle and Dickert, 2007; Rohan, 2000; Sommer, 2016) make it currently difficult to envision 
A/IS that have computational structures directly corresponding to social or cultural values 
(such as “security,” “autonomy,” or “fairness”). However, it is a more realistic goal to embed 
explicit norms into such systems because norms can be considered instructions to act in 
defined ways in defined contexts, for a specific community (from family to town to country 
and beyond). A community's network of norms is likely to reflect the community's values, 
and A/IS equipped with such a network would, therefore, also reflect the community's 
values, even if there are no directly identifiable computational structures that correspond 
to values per se. (For discussion of specific values that are critical for ethical considerations 
of A/IS, see the sections “Personal Data and Individual Access Control" and “Well-being”) 


Norms are typically expressed in terms of obligations and prohibitions, and these can 

be expressed computationally (e.g., Malle, Scheutz, and Austerweil, 2017; Vazquez- 
Salceda, Aldewereld, Dignum, 2004). At this level, norms are typically qualitative in nature 
(e.g., do not stand too close to people). However, the implementation of norms also 

has a quantitative component (the measurement of the physical distance we mean 

by “too close”), and the possible instantiations of the quantitative component technically 
enable the qualitative norm. 
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To address the broad objective of embedding norms and, by implication, values into 
these systems, our Committee has defined three more concrete goals as described 
in the following sections: 


1. Identifying the norms of a specific community in which A/IS operate. 
2. Computationally implementing the norms of that community within the A/IS. 


3. Evaluating whether the implementation of the identified norms in the A/IS 
are indeed conforming to the norms reflective of that community. 


Pursuing these three goals represents an iterative process that is sensitive to the purpose 
of A/IS and their users within a specific community. It is understood that there may be 
clashes of values and norms when identifying, implementing, and evaluating these systems. 
Such clashes are a natural part of the dynamically changing and renegotiated norm systems 
of any community. As a result, we advocate for an approach where systems are designed 
to provide transparent signals (such as explanations or inspection capabilities) about 

the specific nature of their behavior to the individuals in the community they serve. 


References 


e  Hitlin, S., and J. A. Piliavin. “Values: Reviving a Dormant Concept.” Annual Review 
of Sociology 30 (2004): 359-393. 


e Malle, B. F, and S. Dickert. “Values,” The Encyclopedia of Social Psychology, edited 
by R. F. Baumeister and K. D. Vohs, Thousand Oaks, CA: Sage, 2007. 
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Section 1 — Identifying Norms 
for Autonomous Intelligent Systems 


We identify three issues that must be addressed 
in the attempt to identify norms (and thereby 
values) for A/IS. The first issue asks which 

norms should be identified, and with which 
properties. Here we highlight context specificity 
as a fundamental property of norms. Second, 

we emphasize another fundamental property of 
norms: their dynamically changing nature, which 
requires A/IS to have the capacity to update their 
norms and learn new ones. Third, we address the 
challenge of norm conflicts that naturally arise in 
a complex social world. Resolving such conflicts 
requires priority structures among norms, which 
help determine whether, in a given context, 
adhering to one norm is more important than 
adhering to another norm. 





Issue 1: 


Which norms should be 
identified? 


Background and Analysis 


If machines engage in human communities 

as autonomous agents, then those agents will 
be expected to follow the community's social 
and moral norms. A necessary step in enabling 
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machines to do so is to identify these norms. 
But which norms? Laws are publicly documented 
and therefore easy to identify, so they will 
certainly have to be incorporated into A/IS. Social 
and moral norms are more difficult to ascertain, 
as they are expressed through behavior, language, 
customs, cultural symbols, and artifacts. Most 
important, communities (from families to whole 
nations) differ to various degrees in the laws 

and norms they follow. Therefore, generating 

a universal set of norms that applies to all 
autonomous systems is not realistic, but neither 
is it advisable to completely personalize an A/IS 
to individual preferences. However, we believe 
that identifying broadly observed norms of 

a particular community is feasible. 


The difficulty of generating a set of universal 
norms is not inconsistent with the goal of 
seeking agreement over Universal Human Rights 
(see “General Principles” section). However, 
such universal rights would not be sufficient 

for devising an A/IS that obeys the specific 
norms of its community. Universal rights must, 
however, constrain the kinds of norms that are 
implemented in an A/IS. 


Embedding norms in A/IS requires a clear 
delineation of the community in which the 
A/IS are to be deployed. Further, even within 
a particular community, different types of 
A/IS will demand different sets of norms. 
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The relevant norms for self-driving vehicles, for 
example, will differ greatly from those for robots 
used in healthcare. Thus, we recommend that 
to develop A/IS capable of following social and 
moral norms, the first step is to identify the 
norms of the specific community in which the 
A/IS are to be deployed and, in particular, norms 
relevant to the kinds of tasks that the A/IS are 
designed to perform. Even when designating 

a narrowly defined community (e.g., a nursing 
home; an apartment complex; a company), 
there will be variations in the norms that apply. 
The identification process must heed such 
variation and ensure that the identified norms 
are representative not only of the dominant 
subgroup in the community but also of 
vulnerable and underrepresented groups. 


The most narrowly defined community is a 
single person, and A/IS may well have to adapt 
to the unique norms of a given individual, such 
as norms of arranging a disabled person's home 
to accommodate certain physical limitations. 
However, unique individual norms must not 
violate norms in the larger community. Whereas 
the arrangement of someone's kitchen or the 
frequency with which a care robot checks in with 
a patient can be personalized without violating 
any community norms, encouraging the robot 
to use derogatory language to talk about certain 
social groups does violate such norms. (In the 
next section we discuss how A/IS might handle 
such norm conflicts.) 


We should note that the norms that apply to 
humans may not always be identical to the norms 
that would apply to an A/IS in the same context. 


Empirical research involving multiple disciplines 
and multiple methods (see the Further Resources 
section) should therefore (a) investigate and 
document both community- and task-specific 
norms that apply to humans and (b) consider 
possible differences for A/IS deployed in these 
contexts. The set of empirically identified norms 
applicable to A/IS should then be made available 
for designers to implement. 


Candidate Recommendation 


To develop A/IS capable of following social and 
moral norms, the first step is to identify the 
norms of the specific community in which the 
A/IS are to be deployed and, in particular, norms 
relevant to the kinds of tasks that the A/IS are 
designed to perform. 


Further Resources 


e Bendel, O. Die Moral in der Maschine: 
Beiträge zu Roboter- und Maschinenethik. 
Hannover, Germany: Heise Medien, 2016. 
Accessible popular-science contributions 
to philosophical issues and technical 
implementations of machine ethics. 


e Burks, S. V., and E. L. Krupka. “A Multimethod 


Approach to Identifying Norms and 
Normative Expectations within a Corporate 
Hierarchy: Evidence from the Financial 
Services Industry." Management Science 
58 (2012): 203-217. Illustrates surveys 
and incentivized coordination games as 
methods to elicit norms in a large financial 
services firm. 
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* — Friedman, B., P. H. Kahn, A. Borning, and 
A. Huldtgren. “Value Sensitive Design and 
Information Systems,” in Early Engagement 
and New Technologies: Opening up the 
Laboratory (Vol. 16), edited by N. Doorn, 
D. Schuurbiers, |. van de Poel, and M. 
E. Gorman, 55—95. Dordrecht: Springer, 
2013. A comprehensive introduction into 
Value Sensitive Design and three sample 
applications. 


e Mackie, G., F. Moneti, E. Denny, and 
H. Shakya. What Are Social Norms? How 
Are They Measured? UNICEF Working Paper. 
University of California at San Diego: 
UNICEF, 2012. A broad survey of conceptual 
and measurement questions regarding 
social norms. 








e Malle, B. F. “Integrating Robot Ethics and 
Machine Morality: The Study and Design 
of Moral Competence in Robots.” Ethics 
and Information Technology 18, no. 4 
(2016): 243-256. Discusses how a robot's 
norm capacity fits in the larger vision of 
a robot with moral competence. 


e Miller, K. W., M. J. Wolf, and F. Grodzinsky. 
“This ‘Ethical Trap’ Is for Roboticists, Not 
Robots: On the Issue of Artificial Agent Ethical 
Decision-Making.” Science and Engineering 
Ethics 23 (2017): 389—401. This article 
raises doubts about the possibility of imbuing 
artificial agents with morality, or claiming 
to have done so. 
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Rizzo, A., and L. L. Swisher. “Comparing 

the Stewart—Sprinthall Management Survey 
and the Defining Issues Test-2 as Measures 
of Moral Reasoning in Public Administration." 
Journal of Public Administration Research 
and Theory 14 (2004): 335-348. Describes 
two assessment instruments of moral 
reasoning (including norm maintenance) 
based on Kohlberg’s theory of moral 
development. 


Schwartz, S. H. “An Overview of the Schwartz 
Theory of Basic Values.” Online Readings 

in Psychology and Culture 2 (2012). 
Comprehensive overview of a specific 

theory of values, understood as motivational 
orientations toward abstract outcomes 

(e.g., self-direction, power, security). 








Schwartz, S. H., and K. Boehnke. 
“Evaluating the Structure of Human Values 
with Confirmatory Factor Analysis.” Journal 
of Research in Personality 38 (2004): 
230-255. Describes an older method of 
subjective judgments of relations among 
valued outcomes and a newer, 

formal method of analyzing these relations. 











Wallach, W., and C. Allen. Moral Machines: 
Teaching Robots Right from Wrong. 

New York: Oxford University Press, 2008. 
This book describes some of the challenges 
of having a one-size-fits-all approach to 
embedding human values in autonomous 
systems. 
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Issue 2: 
The need for norm updating. 


Background and Analysis 


Norms are not static. They change over time, 

in response to social progress and new legal 
measures, and, in smaller communities, in 
response to complaints or new opportunities. 
New norms form when technological innovation 
demands novel social standards (e.g., cell phone 
use in public), and norms can fade away when, 
for whatever reasons, fewer and fewer people 
adhere to them. 


Humans have many mechanisms available 

to update norms and learn new ones. They 
observe other community members’ behavior 
and are sensitive to collective norm change; 

they explicitly ask about new norms when joining 
new communities (e.g., entering college, a job 

in a new town); and they respond to feedback 
from others when they exhibit uncertainty about 
norms or have violated a norm. 


An A/IS may be equipped with a norm baseline 
before it is deployed in its target community 
(Issue 1), but this will not suffice for it to behave 
appropriately over an extended time. It must 

be capable of identifying and adding new norms 
to its baseline system, because the initial norm 
identification process will undoubtedly have 
missed some norms. It must also be capable 

of updating some of its existing norms, as change 
occurs in its target community. A/IS would be 


best equipped to respond to such demands 
for change by relying on multiple mechanisms, 
such as: 


e — Processing behavioral trends by members 
of the target community and comparing 
them to trends predicted by the baseline 
norm system; 


e — Asking for guidance from the community 
when uncertainty about applicable norms 
exceeds a critical threshold; 


e — Responding to instruction from the 
community members who introduce the 
robot to a previously unknown context 
or who notice the A/IS's uncertainty in 
a familiar context; 


* Responding to critique from the community 
when the A/IS violates a norm. 


The modification of a normative system can 
occur at any level of the system: it could involve 
altering the priority weightings between individual 
norms, (changing the qualitative expression of 

a norm), or altering the quantitative parameters 
that enable the norm. 


As in the case of resolving norm conflicts 
(Issue 2), we recommend that the system's 
norm changes be transparent. That is, the 
system should make explicit when it adds new 
norms to its norm system or adjusts the priority 
or content of existing norms. The specific 

form of communication will vary by machine 
sophistication (e.g, communication capacity) 
and function (e.g., flexible social companion 

vs. task-defined medical robot). In some cases, 
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the system may document its dynamic change 
and the user can consult this documentation as 
desired; in other cases, explicit announcements 
and requests for discussion may be appropriate; 
in yet other cases, the A/IS may propose changes 
and the relevant human community will decide 
whether such changes should be implemented 

in the system. 


Candidate Recommendation 


To respond to the dynamic change of norms 

in society the A/IS must be able to adjust its 
existing norms and learn new ones, while being 
transparent about these changes. 





Issue 3: 


A/IS will face norm conflicts 
and need methods to 
resolve them. 


Background and Analysis 


Often, even within a well-specified context, no 
action is available that fulfills all obligations and 
prohibitions. Such situations (often described as 
moral dilemmas or moral overload; see Van den 
Hoven, 2012) must be computationally tractable 
by an A/IS — it cannot simply stop in its tracks 
and end on a logical contradiction. Humans 
resolve such situations by accepting trade-offs 
between conflicting norms, which constitute 


priorities of one norm or value over another 
(in a given context). Such priorities may be 
represented in the norm system as hierarchical 
relations. 


Along with identifying the norms within a specific 
community and task domain, we need to identify 
the ways in which people prioritize competing 
norms and resolve norm conflicts, and the ways 
in which people expect A/IS to resolve similar 
norm conflicts. Some general principles are 
available, such as the Common Good Principle 
(Andre and Velasquez, 1992). However, other 
priority relations in the norm network must be 
established through empirical research so as to 
reflect the shared values of the community in 
question. For example, a self-driving vehicle's 
prioritization of one factor over another in its 
decision-making will need to reflect the priority 
order of values of its target user population, 

even if this order is in conflict with that of an 
individual designer, manufacturer, or client. 


Some priority orders can be built into a 

given norm network as hierarchical relations 
(e.g., prohibitions against harm to humans 
typically override prohibitions against lying). 
Other priority orders can stem from the general 
override that norms in the larger community 
exert on norms and preferences of an individual 
user. In the earlier example discussing 
personalization (see Issue 1), an A/IS of a racist 
user who demands the A/IS use derogatory 
language for certain social groups might have 
to resist such demands because community 
norms hierarchically override an individual 
user's preferences. 
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In many cases, priority orders are not built in 

as fixed hierarchies because the priorities are 
themselves context specific or may arise from 
net moral costs and benefits of the particular 
case at hand. A/IS must have learning capacities 
to track such variations and incorporate user 
input (e.g., about the subtle differences between 
contexts) to refine the system's norm network 
(see Issue 2). 


We also recommend that the system's resolution 
of norm conflicts be transparent — that is, 
documented by the system and ready to be 
made available to users. Just like people explain 
to each other why they made decisions, they will 
expect any A/IS to be able to explain its decisions 
(and be sensitive to user feedback about the 
appropriateness of the decision). To do so, design 
and development of A/IS should specifically 
identify the relevant groups of humans who 

may request explanations and evaluate the 
system's behavior. 


Candidate Recommendation 


One must identify the ways in which people 
resolve norm conflicts and the ways in which 
they expect A/IS to resolve similar norm conflicts. 
The system's resolution of norm conflicts must 
be transparent — that is, documented by the 
system and ready to be made available to 
relevant users. 
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Section 2 — Implementing Norms 
in Autonomous Intelligent Systems 


Once the norms relevant to an A/IS's role in 

a specific community have been identified, 
including their properties and priority structure, 
we must link these norms to the functionalities of 
the underlying computational system. We discuss 
three issues that arise in this process of norm 
implementation. First, computational approaches 
to enable a system to represent, learn, and 
execute norms are only slowly emerging. 
However, the diversity of approaches may soon 
lead to substantial advances. Second, for A/ 

IS that operate in human communities, there 

is a particular need for transparency — ranging 
from the technical process of implementation 

to the ethical decisions that A/IS will make in 
human-machine interactions, which will require 

a high level of explainability. Third, failures of 
normative reasoning can be considered inevitable 
and mitigation strategies should therefore be 

put in place to handle such failures when they 
occur. Before we discuss these three issues and 
corresponding candidate recommendations, we 
offer one general recommendation for the entire 
process of implementation: 


Candidate Recommendation 


Throughout the technical implementation 

of norms, designers should already consider 
forms and metrics of evaluation and define 

and incorporate central criteria for assessing 

an A/IS's norm conformity (e.g., human-machine 
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agreement on moral decisions, verifiability 
of A/IS decisions, justified trust). 





Issue 1: 


Many approaches to norm 
implementation are currently 
available, and new ones are 
being developed. 


Background and Analysis 


The prospect of developing artificial systems that 
are sensitive to human norms and factor them 
into morally or legally significant decisions has 
intrigued science fiction writers, philosophers, 
and computer scientists alike. Modest efforts 

to realize this worthy goal in limited or bounded 
contexts are already underway. This emerging 
field of research appears under many names, 
including: machine morality, machine ethics, 
moral machines, value alignment, computational 
ethics, artificial morality, safe Al, and friendly Al. 


There are a number of different implementation 
routes for implementing ethics into autonomous 
systems. Following Wallach and Allen (2008), 
we might begin to categorize these as either: 
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A. Top-down approaches, where the system 
(e.g., a software agent) has some symbolic 
representation of its activity, and so can 
identify specific states, plans, or actions as 
ethical/unethical with respect to particular 
ethical requirements (e.g., Dennis, Fisher, 
Slavkovik, Webster, 2016; Pereira and 
Saptawijaya, 2016; Rotzer, 2016; Scheutz, 
Malle, and Briggs, 2015); or 


B. Bottom-up approaches, where the system 
(e.g., a learning component) builds up, 
through experience of what is to be 
considered ethical/unethical in certain 
situations, an implicit notion of ethical 
behavior (e.g., Anderson and Anderson, 
2014; Riedl and Harrison, 2016). 


Relevant examples of these two are: (A) symbolic 
agents that have explicit representations of plans, 
actions, goals, etc.; and (B) machine learning 
systems that train subsymbolic mechanisms with 
acceptable ethical behavior. (For more detailed 
discussion, see Charisi et al., 2017.) 


Computers and robots already reflect values in 
their choices and actions, but these values are 
programmed or designed in by the engineers 
that build the systems. Increasingly, autonomous 
systems will encounter situations that their 
designers cannot anticipate and will require 
algorithmic procedures to select the better 

of two or more possible courses of action. 
Many of the existing experimental approaches 
to building moral machines are top-down, 

in the sense that norms, rules, principles, or 
procedures are used by the system to evaluate 
the acceptability of differing courses of action, 
or as moral standards or goals to be realized. 


Recent breakthroughs in machine learning and 
perception will enable researchers to explore 
bottom-up approaches in which the Al system 
learns about its context and about human 
norms, similar to the manner in which a child 
slowly learns which forms of behavior are safe 
and acceptable. Of course a child can feel pain 
and pleasure, empathize with others, and has 
other capabilities that an Al system cannot 
presently imitate. Nevertheless, as research on 
autonomous systems progresses, engineers 
will explore new ways to either simulate learning 
capabilities or build alternative mechanisms 
that fulfill similar functions. 


Each of the first two options has obvious 
limitations, such as option A's inability to learn 
and adapt and option B's unconstrained learning 
behavior. A third option tries to address these 
limitations: 


C. Hybrid approaches, combining (A) 
and (B). 


For example, the selection of action might be 
carried out by a subsymbolic system, but this 
action must be checked by a symbolic “gateway” 
agent before being invoked. This is a typical 
approach for Ethical Governors (Arkin, 2008; 
Winfield, Blum, and Liu, 2014) or Guardians 
(Etzioni, 2016) that monitor, restrict, and even 
adapt certain unacceptable behaviors proposed 
by the system. (See also Issue 3.) Alternatively, 
action selection in light of norms could be done 
in a verifiable logical format, while many of the 
norms constraining those actions can be learned 
through bottom-up learning mechanisms 

(e.g., Arnold, Kasenberg, and Scheutz, 2017). 
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These three architectures are not a 
comprehensive list of all possible techniques 
for implementing norms and values into A/IS. 
For example, some contributors to the multi- 
agent systems literature have integrated norms 
into their agent specifications (Andrighetto et 
al., 2013), and even though these agents live in 
societal simulations and are too underspecified 
to be translated into individual A/IS (such 

as robots), the emerging work can inform 
cognitive architectures of such A/IS that fully 
integrate norms. In addition, some experimental 
approaches may attempt to capture values 
computationally (Conn, 2017), or attempt 

to relate norms to values in ways that ground 
or justify norms (Sommer, 2016). Of course, 
none of these experimental systems should be 
deployed outside of the laboratory before testing 
or before certain criteria are met, which we 
outline in the remainder of this section and 

in Section 3. 


Candidate Recommendation 


In light of the multiple possible approaches 

to computationally implement norms, diverse 
research efforts should be pursued, especially 
collaborative research between scientists from 
different schools of thought. 
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Issue 2: 


The need for transparency from 
implementation to deployment. 


Background and Analysis 


When A/IS are part of social communities and 
act according to the norms of their communities, 
people will want to understand the A/IS decisions 
and actions, just as they want to understand 
each other's decisions and actions. This is 
particularly true for morally significant actions or 
omissions: an ethical reasoning system should 
be able to explain its own reasoning to a user 

on request. Thus, transparency (or explainability) 
of A/IS is paramount (Wachter, Mittelstadt, and 
Floridi, 2017), and it will allow a community 

to understand, predict, and appropriately trust 
the A/IS (see Section 1, Issue 2). Moreover, as 
the norms embedded in A/IS are continuously 
updated and refined (see Section 1, Issue 2), 
transparency allows for trust to be maintained 
(Grodzinsky, Miller, and Wolf 2011), and, where 
necessary, allows the community to modify 

a system's norms, reasoning, and behavior. 


Transparency can occur at multiple levels 

(e.g., ordinary language, coder verification) and 
for multiple stakeholders (e.g., user, engineer, 
attorney). (See IEEE P7001™, Draft Standard 

for Transparency of Autonomous Systems.) 

It should be noted that transparency to all parties 
may not always be advisable, such as in the 

case of security programs that prevent a system 
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from being hacked (Kroll et al., 2016). Here we 
briefly illustrate the broad range of transparency 
by reference to four ways in which systems 

can be transparent (traceability, verifiability, 
nondeception, and intelligibility) and apply these 
considerations to the implementation of norms 
in A/IS. 


Transparency as traceability. Most relevant for 
the topic of implementation is the transparency 
of the software engineering process during 
implementation (Cleland-Huang, Gotel, and 
Zisman, 2012). It allows for the originally 
identified norms (Section 1, Issue 1) to be 
traced through to the final system. This allows 
technical inspection of which norms have been 
implemented, for which contexts, and how 
norm conflicts are resolved (e.g., priority weights 
given to different norms). Transparency in the 
implementation process may also reveal biases 
that were inadvertently built into systems, such 
as racism and sexism in search engine algorithms 
(e.g., Noble, 2013). (See Section 3, Issue 2.) 
Such traceability in turn calibrates a community's 
trust about whether A/IS are conforming to 

the norms and values relevant in its use context 
(Fleischmann and Wallace, 2005). 


Transparency as verifiability. Transparency 
concerning how normative reasoning is 
approached in the implementation is important 
as we wish to verify that the normative decisions 
the system makes match the required norms and 
values. Explicit and exact representations of these 
normative decisions can then provide the basis 
for a range of strong mathematical techniques, 
such as formal verification (Fisher, Dennis, and 


Webster, 2013). Even if a system cannot explain 
every single reasoning step in understandable 
human terms, a log of ethical reasoning should 
be available for inspection of later evaluation 
purposes. 


Transparency as nondeception and honest 
design. We can assume that lying and deception 
will be prohibited actions in many contexts, 

and therefore will be part of the norm system 
implemented into A/IS. In certain use cases of 
an A/IS, deception may be necessary in serving 
the core functionality of the system (e.g., a robot 
that plays poker with humans), but those actions 
are no longer norm violations because they are 
justified by context and user consent. 


However, the absence of deception does not 
yet meet the goal of transparency. One should 
demand that A/IS be honest, and that includes 
both, more obviously, honest communication 
by the A/IS itself and, less obviously, “honest 
design.” Honest design entails that the physical 
appearance of a system accurately represents 
what the system is capable of doing — e.g., ears 
only for systems that actually process acoustic 
information; eyes only for systems that actually 
process visual information. The requirement 

for honest design may also extend to higher- 
level capacities of artificial agents: If the agent 
introduces a certain topic into conversation, 
then it should also be able to, if asked, reason 
about that topic; if the agent displays signs of a 
certain human-like emotion, then it should have 
an internal state that corresponds to at least an 
analogue to that human emotion (e.g., inhabit 
the appraisal states that make up the emotion). 
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Transparency as intelligibility. As mentioned 
above, humans will want to understand an 
A/IS's decisions and actions, especially the 
morally significant ones. A clear requirement 
for an ethical A/IS is therefore that the system 
be able to explain its own reasoning to a user, 
when asked (or, ideally, also when suspecting 
the user's confusion), and the system should 
do so at a level of ordinary human reasoning, 
not with incomprehensible technical detail 
(Tintarev and Kutlak, 2014). Furthermore, 
when the system cannot itself explain some 
of its actions, technicians or designers should 
be available to make those actions intelligible. 
Along these lines, the European Union's new 
General Data Protection Regulation (GDPR), 
scheduled to take effect in 2018, states that, 
for automated decisions based on personal 
data, individuals have a right to “an explanation 
of the [algorithmic] decision reached after such 
assessment and to challenge the decision.” 
(See Boyd, 2016, for a critical discussion 

of this regulation.) 


Candidate Recommendation 


A/IS, and especially those with embedded 
norms, must have a high level of transparency, 
from traceability in the implementation process, 
mathematical verifiability of its reasoning, 

to honesty in appearance-based signals, and 
intelligibility of the system's operation and 
decisions. 





Issue 3: 
Failures will occur. 


Operational failures and, in particular, violations 
of a system’s embedded community norms are 
unavoidable, both during system testing and 
during deployment. Not only are implementations 
never perfect, but A/IS with embedded norms 
will update or expand their norms over extended 
use (see Section 1, Issue 2) and interactions 

in the social world are particularly complex 

and uncertain. Thus, we propose the following 
candidate recommendation. 


Candidate Recommendation 


Because designers cannot anticipate all possible 
operating conditions and potential failures of 
A/IS, multiple additional strategies to mitigate the 
chance and magnitude of harm must be in place. 


Elaboration 


To be specific, we sample three possible 
mitigation strategies. 


First, anticipating the process of evaluation 
already during the implementation phase requires 
defining criteria and metrics for such evaluation, 
which in turn better allows the detection and 
mitigation of failures. Metrics will include more 
technical variables, such as traceability and 
verifiability; user-level variables such as reliability, 
understandable explanations, and responsiveness 
to feedback; and community-level variables such 
as justified trust (see Issue 2) and the collective 
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belief that A/IS are generally creating social 
benefits rather than, for example, technological 
unemployment. 


Second, a systematic risk analysis and 
management approach can be useful (e.g., 
Oetzel and Spiekermann, 2014, for an application 
to privacy norms). This approach tries to 
anticipate potential points of failure (e.g., norm 
violations) and, where possible, develops some 
ways to mitigate or remove the effects of failures. 
Successful behavior, and occasional failures, 

can then iteratively improve predictions and 
mitigation attempts. 


Third, because not all risks and failures are 
predictable, especially in complex human-machine 
interactions in social contexts, additional mitigation 
mechanisms must be made available. Designers 
are strongly encouraged to augment the 
architectures of their systems with components 
that handle unanticipated norm violations with 

a fail-safe, such as the symbolic “gateway” agents 
discussed in Section 1, Issue 1. Designers should 
identify a number of strict laws (that is, task- and 
communhity-specific norms that should never be 
violated), and the fail-safe components should 
continuously monitor operations against possible 
violations of these laws. In case of violations, 

the higher-order gateway agent should take 
appropriate actions, such as safely disabling 

the system's operation until the source of failure 
is identified. The fail-safe components need to be 
extremely reliable and protected against security 
breaches, which can be achieved, for example, 
by validating them carefully and not letting 

them adapt their parameters during execution. 
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Section 3 — Evaluating 
the Implementation of A/IS 


The success of implementing appropriate 
norms in A/IS must be rigorously evaluated. 
This evaluation process must be anticipated 
during design and incorporated into the 
implementation process, and it must continue 
throughout the life cycle of the system's 
deployment. Assessment before full-scale 
deployment would best take place in systematic 
test beds that allow human users (from the 
defined community, and representing all 
demographic groups) to engage safely with 
the A/IS in intended tasks. Multiple disciplines 
and methods should contribute to developing 
and conducting such evaluations. 


Evaluation criteria must capture the quality of 
human-machine interactions, human approval 
and appreciation of the A/IS, trust in the A/IS, 
adaptability of the A/IS to human users, and 
human benefits in the presence or under 

the influence of the A/IS. A range of ethical/ 
normative aspects to be considered can be found 
in the UK standard on Robot Ethics (BSI, 2016). 
These are important general evaluation criteria, 
but they do not yet fully capture evaluation of 

a system that has norm capacities. To evaluate 

a system's norm-conforming behavior, one must 
describe (and ideally, formally specify) criterion 
behaviors that reflect the previously identified 
norms, describe what the user expects the 
system to do, verify that the system really does 
this, and validate that the specification actually 
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matches the criteria. Many different evaluation 
techniques are available in the field of software 
engineering (Sommerville, 2001), ranging from 
formal mathematical proof, through rigorous 
empirical testing against criteria of normatively 
correct behavior, to informal analysis of user 
interactions and responses to the machine’s 
norm awareness and compliance. All these 
approaches can, in principle, be applied to 

the full range of autonomous systems, including 
robots (Fisher, Dennis, and Webster, 2013). 


Evaluation may be done by first parties 
(designers/manufacturers, and users) as well 
as third parties (e.g., regulators or independent 
testing agencies). In either case, the results 

of evaluations should be made available to 

all parties, with strong encouragement to 
resolve discovered system limitations and 
resolve potential discrepancies among multiple 
evaluations. 


Candidate Recommendation 


Evaluation must be anticipated during a system's 
design, incorporated into the implementation 
process, and continue throughout the system's 
deployment. Evaluation must include multiple 
methods, be made available to all parties 

(from designers and users to regulators), and 
should include procedures to resolve conflicting 
evaluation results. 
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Issue 1: 


Not all norms of a target 
community apply equally 
to human and artificial agents. 


Background and Analysis 


An intuitive criterion for evaluations of norms 
embedded in A/IS would be that the A/IS norms 
should mirror the community's norms — that is, 
the A/IS should be disposed to behave the same 
way that people expect each other to behave. 
However, for a given community and a given 
A/IS use context, A/IS and humans may not have 
identical sets of norms. People will have some 
unique expectations for humans than they do for 
machines (e.g., norms governing the regulation 
of negative emotions, assuming that machines 
do not have such emotions), and people will 
have some unique expectations of A/IS that they 
do not have for humans (e.g., that the machine 
will sacrifice itself, if it can, to prevent harm to 

a human). 


Candidate Recommendation 


The norm identification process must document 
the similarities and differences between the norms 
that humans apply to other humans and the 
norms they apply to A/IS. Norm implementations 
should be evaluated specifically against the norms 
that the community expects the A/IS to follow. 


Embedding Values into Autonomous Intelligent Systems 





Issue 2: 


A/IS can have biases that 
disadvantage specific groups. 


Background and Analysis 


Even when reflecting the full system of 
community norms that was identified, A/IS may 
show operation biases that disadvantage specific 
groups in the community or instill biases in users 
by reinforcing group stereotypes. A system's 

bias can emerge in perception (e.g., a passport 
application Al rejected an Asian man’s photo 
because it insisted his eyes were closed; Griffiths, 
2016); information processing (e.g., speech 
recognition systems are notoriously less accurate 
for female speakers than for male speakers; 
Tatman, 2016); decisions (e.g., a criminal risk 
assessment device overpredicts recidivism 

by African Americans; Angwin, et al., 2016); 

and even in its own appearance and presentation 
(e.g., the vast majority of humanoid robots 

have white “skin” color and use female voices) 
(Riek and Howard, 2014). 


The norm identification process detailed in 
Section 1 is intended to minimize individual 
designers’ biases, because the community 

norms are assessed empirically. The process also 
seeks to incorporate values and norms against 
prejudice and discrimination. However, biases 
may still emerge from imperfections in the norm 
identification process itself, from unrepresentative 
training sets for machine learning systems, and 
from programmers’ and designers’ unconscious 
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assumptions. Therefore, unanticipated or 
undetected biases should be further reduced 

by including members of diverse social groups 

in both the planning and evaluation of Al systems 
and integrating community outreach into the 
evaluation process (e.g., DO-IT program; RRI 
framework). Behavioral scientists and members 
of the target populations will be particularly 
valuable when devising criterion tasks for system 
evaluation. Such tasks would assess, for example, 
whether the A/IS applies norms in discriminatory 
ways to different races, ethnicities, genders, ages, 
body shapes, or to people who use wheelchairs 
or prosthetics, and so on. 


Candidate Recommendation 


Evaluation of A/IS must carefully assess 
potential biases in the system's performance 
that disadvantage specific social groups. 

The evaluation process should integrate 
members of potentially disadvantaged 
groups to diagnose and correct such biases. 





Issue 3: 


Challenges to evaluation 
by third parties. 


Background and Analysis 


A/IS should have sufficient transparency 

to allow evaluation by third parties, including 
regulators, consumer advocates, ethicists, 
post-accident investigators, or society at large. 


However, transparency can be severely limited 

in some systems, especially in those that rely 

on machine learning algorithms trained on large 
data sets. The data sets may not be accessible 
to evaluators; the algorithms may be proprietary 
information or mathematically so complex that 
they defy common-sense explanation; and 

even fellow software experts may be unable 

to verify reliability and efficacy of the final system 
because the system's specifications are opaque. 


For less inscrutable systems, numerous 
techniques are available to evaluate the 
implementation of an A/IS's norm conformity. 
On one side there is formal verification, 

which provides a mathematical proof that 

the A/IS will always match specific normative 
and ethical requirements (typically devised in 
a top-down approach; see Section 2, Issue 1). 
This approach requires access to the decision- 
making process and the reasons for each 
decision (Fisher, Dennis, and Webster, 2013). 
A simpler alternative, sometimes suitable even 
for machine learning systems, is to test the 
A/IS against a set of scenarios and assess how 
well it matches its normative requirements 
(e.g., acting in accordance with relevant norms; 
recognizing other agents’ norm violations). 


These different evaluation techniques can be 
assigned different levels of “strength” — strong 
ones demonstrate the exhaustive set of an 
A/I\S's allowable behaviors for a range of criterion 
scenarios; weaker ones sample from criterion 
scenarios and illustrate the system's behavior 

for that subsample. In the latter case, confidence 
in the A/IS's ability to meet normative 
requirements is more limited. An evaluation’s 
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concluding judgment must therefore 
acknowledge the strength of the verification 
technique used, and the expressed confidence 
in the evaluation (and in the A/IS itself) must 
be qualified by this level of strength. 


Transparency is only a necessary requirement 
for a more important long-term goal, having 
systems be accountable to their users and 
community members. However, this goal raises 
many questions such as to whom the A/IS are 
accountable and who has the right to correct 
the systems, or also which kind of A/IS should 
be subject to accountability requirements. 


Candidate Recommendation 


To maximize effective evaluation by third parties 
(e.g., regulators, accident investigators), A/IS 
should be designed, specified, and documented 
so as to permit the use of strong verification 
and validation techniques for assessing the 
system's safety and norm compliance, in order 
to possibly achieve accountability to the relevant 
communities. 
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To ensure autonomous and intelligent systems (A/IS) are aligned to benefit humanity A/IS 
research and design must be underpinned by ethical and legal norms as well as methods. 
We strongly believe that a value-based design methodology should become the essential 

focus for the modern A/IS organization. 


Value-based system design methods put human advancement at the core of A/IS 
development. Such methods recognize that machines should serve humans, and not the 
other way around. A/IS developers should employ value-based design methods to create 
sustainable systems that are thoroughly scrutinized for social costs and advantages that 
will also increase economic value for organizations. To create A/IS that enhances human 
well-being and freedom, system design methodologies should also be enriched by putting 
greater emphasis on internationally recognized human rights, as a primary form of human 
values. 


To help achieve these goals, researchers and technologists need to embrace transparency 
regarding their processes, products, values, and design practices to increase end-user 

and community trust. It will be essential that educational institutions inform engineering 
students about ethics, justice, and human rights, address ethical research and business 
practices surrounding the development of A/IS, and attend to the responsibility of the 
technology sector vis-a-vis public interest issues. The proliferation of value-based design 
will require a change of current system development approaches for organizations, 
including a commitment of research institutions to strong ethical guidelines for research, 
and of businesses to values that transcend narrow economic incentives. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — Interdisciplinary 
Education and Research 


Integrating applied ethics into education and 
research to address the issues of autonomous 
and intelligent systems (A/IS) requires an 
interdisciplinary approach, bringing together 
humanities, social sciences, science, engineering, 
and other disciplines. 





Issue: 


Inadequate integration 
of ethics in A/IS-related 
degree programs. 


Background 


Al engineers and design teams too often fail 

to thoroughly explore the ethical considerations 
implicit in their technical work and design 
choices. They tend to treat ethical decision- 
making as another form of technical problem 
solving. Although ethical challenges often have 
technical solutions, identifying and ameliorating 
those challenges requires technicians to 
methodically inquire about the social context 
of their work. Moreover, technologists often 
struggle with the imprecision and ambiguity 
inherent in ethical language, which cannot 

be readily articulated and translated into the 
formal languages of mathematics and computer 
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programming associated with algorithms and 
machine learning. Thus, ethical issues can easily 
be rendered invisible or inappropriately reduced 
and simplified in the context of technical practice. 
This originates in the fact that many engineering 
programs do not sufficiently integrate coursework, 
training, or practical experience in applied ethics 
throughout their curricula; too often ethics is 
relegated to a stand-alone course or module 

that gives students little or no direct experience 
in ethical decision-making in engineering work. 
Ethics education for engineering students should 
be meaningful, measurable, and incorporate best 
practices of STEM ethics education drawn from 
pertinent multidisciplinary resources. 


The aim of these recommendations is to 
prepare students for the technical training and 
engineering development methodologies that 
incorporate ethics as essential so that ethics 
and human rights become naturally part of 
the design process. 


Candidate Recommendations 


Ethics and ethical reflection need to be a core 
subject for aspiring engineers and technologists 
beginning at the earliest appropriate level and 
for all advanced degrees. By training students 
how to be sensitive to ethical issues in design 
before they enter the workplace, they can 
more effectively implement value-based design 
methodologies in the context of A/IS work. 
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We also recommend that effective STEM ethics 
curricula be informed by scientists, artists, 
philosophers, psychologists, legal scholars, 
engineers, and other subject matter experts from 
a variety of cultural backgrounds to ensure that 
students acquire sensitivity to a diversity of robust 
perspectives on human flourishing. Such curricula 
should teach aspiring engineers, computer 
scientists, and statisticians about the relevance 
and impact of their decisions in designing A/ 

IS technologies. Effective ethics education in 
STEM contexts should span primary, secondary, 
and post-secondary education, and include 

both universities and vocational training schools. 
Relevant accreditation bodies should reinforce 
this integrated approach as outlined above. 


Further Resources 


e Holdren, J., and M. Smith. “Preparing for the 
Future of Artificial Intelligence.” Washington, 
DC: Executive Office of the President, 
National Science and Technology Council, 
2016. This White House report makes several 
recommendations on how to ensure that 
Al practitioners are aware of ethical issues 
by providing them with ethical training. 





* The French Commission on the Ethics 
of Research in Digital Sciences and 
Technologies (CERNA) recommends 
including ethics classes in doctoral programs. 











e The U.S. National Science Foundation 
has funded extensive research on STEM 
ethics education best practices through 
the Cultivating Cultures for Ethical Science, 
Technology, Engineering, and Mathematics 
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(CCE-STEM) Program, and recommends 
integrative approaches that incorporate 
ethics throughout STEM education. 





Comparing the UK, EU, and US approaches 
to Al and ethics: Cath, C. et al. “Artificial 
Intelligence and the ‘Good Society’: 

The US, EU, and UK Approach.” Science 
and Engineering Ethics (2017). 











The Oxford Internet Institute (Oll) organized 
a workshop on ethical issues in engineering. 
The output paper can be found here: 
Zevenbergen, B. et al. “Philosophy Meets 
Internet Engineering: Ethics in Networked 
Systems Research.” Oxford, U.K.: Oxford 
Internet Institute, University of Oxford, 2015. 











Companies should also be encouraged 
to mandate consideration of ethics at the 
pre-product design stage, as was done 
by Lucid Al. 


There are a variety of peer-reviewed online 
resources collecting STEM ethics curricula, 
syllabi, and education modules: 


* Ethics Education Library, Illinois Institute 
of Technology 





e — IDEESE: International Dimensions of 
Ethics Education in Science & Engineering, 
University of Massachusetts Amherst 











e National Center for Professional & 
Research Ethics, University of Illinois 








* Online Ethics Center, National Academy 
of Engineering 





This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 57 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Methodologies to Guide Ethical Research and Design 








Issue: 


The need for more constructive 
and sustained interdisciplinary 
collaborations to address ethical 
issues concerning autonomous 
and intelligent systems (A/IS). 


Background 


Not enough institutional resources and incentive 
structures exist for bringing A/IS engineers 

and designers into sustained and constructive 
contact with ethicists, legal scholars, and social 
scientists, both in academia and industry. This 
contact is necessary as it can enable meaningful 
interdisciplinary collaboration to shape the future 
of technological innovation. There are currently 
few methodologies, shared knowledge, and 
lexicons that would facilitate such collaborations. 


This issue, to a large degree, relates to funding 
models as well as the traditional mono-function 
culture in A/IS-related institutions and companies, 
which limit cross-pollination between disciplines 
(see below). To help bridge this gap, additional 
“translation work" and resource sharing (including 
websites and MOOCs) needs to happen among 
technologists and other relevant experts (e.g., 

in medicine, architecture, law, philosophy, 
psychology, cognitive science). 


Candidate Recommendations 


Funding models and institutional incentive 
structures should be reviewed and revised to 
prioritize projects with interdisciplinary ethics 
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components to encourage integration of ethics 
into projects at all levels. 


Further Resources 


e Barocas, S. Course Material for Ethics and 
Policy in Data Science. 








e Floridi, L., and M. Taddeo. “What Is Data 
Ethics?” Philosophical Transactions of 
the Royal Society 374, no. 2083 (2014): 
1—4. doi:10.1098/rsta.2016.0360. 





* Spiekermann, S. Ethical IT Innovation: A 
Value-Based System Design Approach. Boca 
Raton, Florida: Auerbach Publications, 2015. 


e The approach developed by the Internet 
Research Task Force's Human Rights Protocol 


Research Group (HRPC) for integrating 
human rights concern in technical design. 











Issue: 


The need to differentiate 
culturally distinctive values 
embedded in Al design. 


Background 


A responsible approach to embedded values 
(both as uncritical bias and as value by design) 
in information and communications technology 
(ICT), algorithms and autonomous systems 

will need to differentiate between culturally 
distinctive values (i.e., how do different cultures 
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view privacy, or do they at all? And how do 

these differing presumptions of privacy inform 
engineers and technologists and the technologies 
designed by them?). Without falling into 
oversimplified ethical relativism, or embedding 
values that are antithetical to human flourishing 
(for example, human rights violations), it is 
critical that A/IS design avoids only considering 
monocultural influenced ethical foundations. 


Candidate Recommendations 


Establish a leading role for intercultural 
information ethics (IIE) practitioners in ethics 
committees informing technologists, policy 
makers, and engineers. Clearly demonstrate 
through examples how cultural bias informs not 
only information flows and information systems, 
but also algorithmic decision-making and value 
by design. 
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Further Resources 


Pauleen, D. J. et al. “Cultural Bias in 
Information Systems Research and Practice: 
Are You Coming From the Same Place | 
Am?” Communications of the Association 
for Information Systems 17, no. 17 (2006). 
The work of Pauleen et al. (2006) and 
Bielby (2015) has been guiding in this field: 
“Cultural values, attitudes, and behaviours 
prominently influence how a given group 
of people views, understands, processes, 
communicates, and manages data, 
information, and knowledge.” 








Bielby, J. “Comparative Philosophies in 
Intercultural Information Ethics,” Confluence: 
Online Journal of World Philosophies 2, 

no. 1 (2015): 233-253. 
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Section 2 — Corporate Practices 
and A/IS 


Corporations, whether for-profit or not-for-profit, Candidate Recommendations 
are eager to develop, deploy, and monetize 

A/IS, but there are insufficient structures in place 
for creating and supporting ethical systems and 
practices around A/IS funding, development, 


The building blocks of such practices include 
top-down leadership, bottom-up empowerment, 
ownership, and responsibility, and the need to 
consider system deployment contexts and/or 





Or USE: ecosystems. The institution of an ethical A/IS 
corporate culture would accelerate the adoption 
of the other recommendations within this section 
focused on business practices. 

Issue: 


P Further Resources 
Lack of value-based ethical 
e The website of the Benefit corporations 


culture and practices (B-corporations) provides a good overview 


for industry. of a range of companies that personify this 
type of culture. 





e — Firms of Endearment is a book which 











Background 
showcases how companies embracing 
There is a need to create value-based ethical values and a stakeholder approach 
culture and practices for the development and outperform their competitors in the 
deployment of products based on autonomous long run. 
and intelligent systems (A/IS). To do so, we need 
to further identify and refine social processes * The ACM Code of Ethics and Professional 
and management strategies that facilitate Ethics, which also includes various 
values-based design in the engineering and references to human well-being and 
manufacturing process. human rights. 
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Lack of values-aware 
leadership. 


Background 


Technology leadership should give innovation 
teams and engineers direction regarding which 
human values and legal norms should be 
promoted in the design of an A/IS system. 
Cultivating an ethical corporate culture is an 
essential component of successful leadership 
in the A/IS domain. 


Candidate Recommendations 


Companies need to create roles for senior- 

level marketers, ethicists, or lawyers who can 
pragmatically implement ethically aligned 
design, both the technology and the social 
processes to support value-based system 
innovation. Companies need to ensure that their 
understanding of value-based system innovation 
is based on de jure and de facto international 
human rights standards. 


A promising way to ensure values are on the 
agenda in system development is to have 

a Chief Values Officer (CVO), a role first 
suggested by Kay Firth-Butterfield, Vice-Chair, 
The IEEE Global Initiative and Project Head of 
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Al and Machine Learning at the World Economic 
Forum. The CVO should support system 
innovations and engineering teams to consider 
values and provide them with methodological 
guidance on how to do so. However, ethical 
responsibility should not be delegated solely to 
CVOs. CVOs can support the creation of ethical 
knowledge in companies, but in the end all 
members of an innovation team will need to 
act responsibly throughout the design process. 


Further Resources 


* United Nations, Guiding Principles on 
Business and Human Rights: Implementing 
the United Nations “Protect, Respect and 
Remedy” Framework, New York and Geneva: 
UN, 2011. 














e Institute for Human Rights and Business 
(IHRB), and Shift, SectlCTor Guide on 
Implementing the UN Guiding Principles 
on Business and Human Rights, 2013. 


e Cath, C., and L. Floridi. “The Design of 
the Internet's Architecture by the Internet 
Engineering Task Force (IETF) and Human 
Rights.” Science and Engineering Ethics 23, 
no. 2 (2017): 449-468. 








e Butterfield, Kay-Firth (2017). How IEEE 
Aims to Instill Ethics in Artificial Intelligence 
Design. The Institute. 
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Lack of empowerment 
to raise ethical concerns. 


Background 


Engineers and design teams can encounter 
obstacles to raising ethical concerns regarding 
their designs or design specifications within 
their organizations. Corporate culture should 
incentivize technical staff to voice the full range 
of ethical questions to relevant corporate actors 
throughout the full product lifecycle. Because 
raising ethical concerns can be perceived as 
slowing or halting a design project, organizations 
need to consider how they can recognize and 
incentivize value-based design as an integral 
component of product development. 


Candidate Recommendations 


Employees should be empowered to raise ethical 
concerns in day-to-day professional practice, 

not just in extreme emergency circumstances 
such as whistleblowing. New organizational and 
socio-cultural processes that broaden the scope 
around professional ethics and design need 

to be implemented within organizations. New 
categories of considerations around these issues 
need to be accommodated along with new 
forms of Codes of Conduct, so individuals are 
empowered to share their insights and concerns 
in an atmosphere of trust. 


Further Resources 


* The British Computer Society (BCS) code 
of conduct holds that individuals have 
to: “a) have due regard for public health, 
privacy, security and well-being of others 
and the environment. b) have due regard 
for the legitimate rights of Third Parties. 
c) conduct your professional activities 
without discrimination on the grounds 
of sex, sexual orientation, marital status, 
nationality, colour, race, ethnic origin, religion, 
age or disability, or of any other condition 
or requirement. d) promote equal access 
to the benefits of IT and seek to promote 
the inclusion of all sectors in society 
wherever opportunities arise.” 





+ The Design of the Internet's Architecture 
by the Internet Engineering Task Force 
(IETF) and Human Rights mitigates the 
issue surrounding the lack of empowerment 
to raise ethical concerns as they relate to 
human rights by suggesting that companies 
can implement measures that emphasize 
responsibility-by-design. This term refers 
to solutions where the in-house working 
methods ensure that engineers have 
thought through the potential impact of their 
technology, where a responsible attitude 
to design is built into the workflow. 
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Issue: 


Organizations should examine 
their cultures to determine 
how to flexibly implement 
value-based design. 


Background 


Ethics is often treated as an impediment to 
innovation, even among those who ostensibly 
support ethical design practices. In industries 
that reward rapid innovation, it is necessary to 
develop design practices that integrate effectively 
with existing engineering workflows. Those who 
advocate for ethical design within a company 
should not be seen as innovators seeking the 
best ultimate outcomes for the company, end- 
users, and society. Leaders can facilitate that 
mindset by promoting an organizational structure 
that supports the integration of dialogue about 
ethics throughout product lifecycles. 


A/IS design processes often present moments 
where ethical consequences can be highlighted. 
There are no universally prescribed models 

for this because organizations vary significantly 
in structure and culture. In some organizations, 
design team meetings may be brief and informal. 
In others, the meetings may be lengthy and 
structured. Regardless, team members should 
understand how to raise such questions without 
being perceived as impediments by peers 

and managers. The transitions point between 
discovery, prototyping, release, and revisions 

are natural contexts for conducting such reviews. 
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Iterative review processes are also advisable, in 
part because changes to risk profiles over time 
can illustrate needs or opportunities for improving 
the final product. 


Candidate Recommendations 


Companies should study their own design 
processes to identify moments where engineers 
and researchers can be encouraged to raise 
and resolve questions of ethics. Achieving 

a distributed responsibility for ethics requires 
that all people involved in product design are 
encouraged to notice and respond to ethical 
concerns, particularly around safety, bias, and 
legality. Organizations should consider how 
they can best encourage and accommodate 
lightweight deliberations among peers. 


Additionally, organizations should identify 
points for formal review inside their product 
development processes. These reviews can 
focus on “red flags” that have been identified 
in advance as indicators of risk. For example, 

if the datasets involve minors or focus on users 
from protected classes then it may require 
additional justification or alterations to the 
research or development protocols. 


Further Resources 


* — Sinclair, A. “Approaches to Organizational 
Culture and Ethics.” Journal of Business 
Ethics 12, no. 1 (1993): 63-73. 








e Chen, A. Y. S., R. B. Sawyers, and P. F. 
Williams. “Reinforcing Ethical Decision Making 
Through Corporate Culture.” Journal of 
Business Ethics 16, no. 8 (1997): 855-865. 
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e Crawford, K., and R. Calo. “There Is a Blind 
Spot in Al Research.” Nature 538 (2016): 
411-313. 





Issue: 


Lack of ownership or 
responsibility from the tech 
community. 


Background 


There is a divergence between the values the 
technology community sees as its responsibility 
in regards to A/IS, and the broader set of 

social concerns raised by the public, legal, and 
professional communities. The current makeup 
of most organizations has clear delineations 
among engineering, legal, and marketing 

arenas. Thus technologists feel responsible for 
safety issues regarding their work, but for larger 
social issues may say, “legal will handle that.” 

In addition, in employment and management 
technology or work contexts, “ethics” typically 
refers to a code of conduct regarding professional 
decorum (versus a values-driven design process 
mentality). As such, ethics regarding professional 
conduct often implies moral issues such as 
integrity or the lack thereof (in the case of 
whistleblowing, for instance), but ethics in A/IS 
design includes broader considerations about 
the consequences of technologies. 
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Candidate Recommendations 


To help integrate applied ethics regarding 

A/IS and in general, organizations need to choose 
specific language that will break down traditional 
biases or barriers and increase adoption of 
values-based design. For instance, an organization 
can refer to the “trade-offs” (or “value trade- 
offs") involved in the examination of the fairness 
of an algorithm to a specific end user population. 


Organizations should clarify the relationship 
between professional ethics and applied 

A/IS ethics and help designers, engineers, and 
other company representatives discern the 
differences between them and where they 
complement each other. 


Corporate ethical review boards, or comparable 
mechanisms, should be formed to address 
ethical concerns in relation to their A/IS research. 
Such boards should seek an appropriately diverse 
composition and use relevant criteria, including 
both research ethics and product ethics at the 
appropriate levels of advancement of research 
and development. These boards should examine 
justifications of research or industrial projects 

in terms of consequences for human flourishing. 


Further Resources 


e Evolving the IRB: Building Robust Review 
for Industry Research by Molly Jackman 
of Facebook explains the differences 
between top-down and bottom up approach 
to the implementation of ethics within 
an organization and describes Facebook's 
internal ethics review for research and 
development. 
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* The article by van der Kloot Meijburg and 
ter Meulen gives a good overview of some 
of the issues involved in “developing 
standards for institutional ethics committees.” 
It focuses specifically on health care 
institutions in the Netherlands, but the 
general lessons drawn can also be applied 
to ethical review boards. Examples of 
organizations dealing with such trade-offs 
can for instance be found in the security 
considerations of the Internet Engineering 
Task Force (IETF). 


Issue: 


Need to include stakeholders 
for adequate ethical 
perspective on A/IS. 


Background 


The interface between Al and practitioners, 

as well as other stakeholders, is gaining 
broader attention in domains such as health 
care diagnostics, and there are many other 
contexts where there may be different levels 

of involvement with the technology. We should 
recognize that, for example, occupational 
therapists and their assistants may have on-the- 
ground expertise in working with a patient, who 
themselves might be the “end user” of a robot 
or social Al technology. Technologists need to 
have that stakeholder feedback, because beyond 
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academically oriented language about ethics, 
that feedback is often about crucial design detail 
gained by experience (form, sound, space, 
dialogue concepts). There are successful models 
of user experience (UX design) that account for 
human factors which should be incorporated 

to A/IS design as systems are more widely 
deployed. 


Candidate Recommendations 


Account for the interests of the full range of 
stakeholders or practitioners who will be working 
alongside A/IS, incorporating their insights. 

Build upon, rather than circumvent or ignore, 

the social and practical wisdom of involved 
practitioners and other stakeholders. 


Further Resources 


* Schroeter, Ch. et al. “Realization and User 
Evaluation of a Companion Robot for 
People with Mild Cognitive Impairments.” 
Proceedings of IEEE International Conference 
on Robotics and Automation (ICRA 2013), 
Karlsruhe, Germany (2013): 1145-1151. 











e Chen, T. L. et al. “Robots for Humanity: 
Using Assistive Robotics to Empower 
People with Disabilities.” /EEE Robotics and 
Automation Magazine 20, no. 1 (2013): 
30-39. 











e Hartson, R., and P. S. Pyla. The UX Book: 
Process and Guidelines for Ensuring a 
Quality User Experience. Waltham, MA: 
Elsevier, 2012. 
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Section 3 — Research Ethics 
for Development and Testing 


of A/IS Technologies 





Issue: 


Institutional ethics committees 
are under-resourced to 
address the ethics of R&D 

in the A/IS fields. 


Background 


It is unclear how research on the interface 

of humans and A/IS, animals and A/IS, and 
biological hazards will pose practical challenges 
for research ethical review boards. Norms, 
institutional controls, and risk metrics appropriate 
to the technology are not well established in 
the relevant literature and research governance 
infrastructure. Additionally, national and 
international regulations governing review 

of human-subjects research may explicitly 

or implicitly exclude A/IS research from their 
purview on the basis of legal technicalities 

or medical ethical concerns regardless 

of potential harms posed by the research. 


Research on A/IS human-machine interaction, 
when it involves intervention or interaction with 
identifiable human participants or their data, 


typically falls to the governance of research 
ethics boards (e.g., institutional review boards). 
The national level and institutional resources 
(e.g., hospitals and universities) to govern ethical 
conduct of HCI, particularly within the disciplines 
pertinent to A/IS research, are underdeveloped. 
First, there is limited international or national 
guidance to govern this form of research. While 
sections of IEEE standards governing research 
on Al in medical devices address some of the 
issues related to security of Al-enabled devices, 
the ethics of testing those devices to bring them 
to market are not developed into recognized 
national (e.g., U.S. FDA) or international 

(e.g., EU EMA) policies or guidance documents. 
Second, the bodies that typically train individuals 
to be gatekeepers for the research ethics bodies 
(e.g, PRIM&R, SoCRA) are under-resourced in 
terms of expertise for A/IS development. Third, 
it is not clear whether there is sufficient attention 
paid to A/IS ethics by research ethics board 
members or by researchers whose projects 
involve the use of human participants or their 
identifiable data. 


Research pertinent to the ethics governing 
research at the interface of animals and 

A/IS research is underdeveloped with respect to 
systematization for implementation by 
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IACUC or other relevant committees. In institutions Further Resources 
without a veterinary school, it is unclear that the 
organization would have the relevant resources 

necessary to conduct an ethical review of such 

research. 


Jordan, S. R. “The Innovation Imperative.” 
Public Management Review 16, no. 1 
(2014): 67-89. 


* Schneiderman, B. “The Dangers of Faulty, 
Biased, or Malicious Algorithms Requires 
Independent Oversight.” Proceedings 
of the National Academy of Sciences of 
the United States of America 113, no. 48 
(2016): 13538-13540. 





Research pertinent to the intersection of 
radiological, biological, and toxicological research 
(ordinarily governed under institutional biosafety 
committees) and A/IS research is not found 
often in the literature pertinent to research 
ethics or research governance. Beyond a limited 














number of pieces addressing the “dual use” or e Metcalf, J., and K. Crawford. “Where Are 
import/export requirements for A/IS in weapons Human Subjects in Big Data Research? 
development, there are no guidelines or The Emerging Ethics Divide.” SSRN Scholarly 
standards governing topics ordinarily reserved Paper, Rochester, NY: Social Science 

for review by institutional biosafety committees, Research Network, 2016. 


or institutional radiological safety committees, 


or laboratory safety committees. e Calo, R. “Consumer Subject Review Boards: 


A Thought Experiment.” Stanford Law 


. 7 Review Online 66 (2013): 97. 
Candidate Recommendations puei Onine BEL ) 


IEEE should draw upon existing standards, 
empirical research, and expertise to identify 
priorities and develop standards for governance 
of A/IS research and to partner with relevant 
national agencies, and international organizations, 
when possible. 
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Section 4 — Lack of Transparency 


Lack of transparency about the A/IS 
manufacturing process presents a challenge 
to ethical implementation and oversight. 





Issue: 


Poor documentation hinders 
ethical design. 


Background 


The limitations and assumptions of a system 
are often not properly documented. Oftentimes 


it is even unclear what data is processed or how. 


Candidate Recommendation 


Software engineers should be required to 
document all of their systems and related data 
flows, their performance, limitations, and risks. 
Ethical values that have been prominent in the 
engineering processes should also be explicitly 
presented as well as empirical evidence of 
compliance and methodology used, such as 
data used to train the system, algorithms and 
components used, and results of behavior 
monitoring. Criteria for such documentation 
could be: auditability, accessibility, 
meaningfulness, and readability. 
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Further Resources 


Cath, C. J. N., L. Glorioso, and M. R. Taddeo. 
“NATO CCD COE Workshop on ‘Ethics 

and Policies for Cyber Warfare’” NATO 
Cybersecurity Centre for Excellence 
(CCDCOE) Report. Oxford, U.K.: Magdalen 
College. Addressed indicators of transparency 
along these lines. 





Turilli, M., and L. Floridi. “The Ethics of 
Information Transparency.” Ethics and 
Information Technology 11, no. 2 (2009): 
105-112. 





Wachter, S., B. Mittelstadt, and L. Floridi. 
“Transparent, Explainable, and Accountable 
Al for Robotics.” Science Robotics 2, no. 6 
(2017). 





Kroll, J. A., J. Huey, S. Barocas, E. W. Felten, 
J. R. Reidenberg, D. G. Robinson, and H. 

Yu. “Accountable Algorithms.” University of 
Pennsylvania Law Review 165, no. 1 (2017): 
633-705. 








Balkin, J. M., “Free Speech in the Algorithmic 
Society: Big Data, Private Governance, and 
New School Speech Regulation.” UC Davis 
Law Review, (2018 forthcoming). 
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Issue: 


Inconsistent or lacking oversight 
for algorithms. 


The algorithms behind intelligent or autonomous 
systems are not subject to consistent oversight. 
This lack of transparency causes concern because 
end users have no context to know how a certain 
algorithm or system came to its conclusions. 
These recommendations are similar to those 
made in committees 1 and 2, but here are used 
as they apply to the narrow scope of this group. 


Candidate Recommendations 


Accountability 


As touched on in the General Principles 

section of Ethically Aligned Design, algorithmic 
transparency is an issue of concern. It is 
understood that specifics relating to algorithms 
or systems contain intellectual property that 
cannot be released to the general public. 
Nonetheless, standards providing oversight of 
the manufacturing process of intelligent and 
autonomous technologies need to be created 

to avoid harm and negative consequences of 
the use of these technologies. Here we can look 
to other technical domains, such as biomedical, 
civil, and aerospace engineering, where 
commercial protections for proprietary technology 
are routinely and effectively balanced with the 
need for appropriate oversight standards and 
mechanisms to safeguard the public. 
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Further Resources 


Frank Pasquale, Professor of Law at the 
University of Maryland, provides the 
following insights regarding accountability 

in a February, 2016 post for the Media 
Policy Project Blog produced by The London 
School of Economics and Political Science. 





Ryan Calo, Associate Professor of Law at 

the University of Washington, wrote an 
excellent article that gives a detailed overview 
of a broad array of Al policy questions. 


In the United States, a recent court case, 
Armstrong, highlights the need for appropriate 
oversight of algorithmic decision-making, 

to preserve due process and other legal 

and ethical principles. K.W. v. Armstrong, 
180 F. Supp. 3d 703 (D. Idaho 2016). 

In the case, a court ruled that Idaho's 
Department of Health and Welfare violated 
the rights of disabled Medicaid recipients by 
relying upon arbitrary and flawed algorithmic 
decision systems when cutting benefits, 

and refusing to disclose the decision bases 
as ‘trade secrets.’ See details of the case 
here: https://www.aclu.org/news/federal- 
court-rules-against-idaho-department- 
health-and-welfare-medicaid-class-action 
and a related discussion of the general risks 
of opaque algorithmic bureaucracies here: 
https://medium.com/aclu/pitfalls-of-artificial- 
intelligence-decisionmaking-highlighted-in- 
idaho-aclu-case-ec59941fb026 
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Issue: 


Lack of an independent 
review organization. 


Background 


We need unaffiliated, expert opinions that 
provide guidance to the general public regarding 
automated and intelligent systems. Currently, 
there is a gap between how A/IS are marketed 
and their actual performance, or application. 
We need to ensure that A/IS technology is 
accompanied by best use recommendations, 
and associated warnings. Additionally, we need 
to develop a certification scheme for A/IS 

that ensures that the technologies have been 
independently assessed as being safe and 
ethically sound. 


For example, today it is possible for systems 
to download new self-parking functionality to 
cars, and no independent reviewer establishes 
or characterizes boundaries or use. Or, when 
a companion robot like Jibo promises to watch 
your children, there is no organization that 

can issue an independent seal of approval or 
limitation on these devices. We need a ratings 
and approval system ready to serve social/ 
automation technologies that will come online 
as soon as possible. We also need further 
government funding for research into how 
A/IS technologies can best be subjected 

to review, and how review organizations can 
consider both traditional health and safety 
issues, as well as ethical considerations. 
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Candidate Recommendations 


An independent, internationally coordinated 
body should be formed to oversee whether such 
products actually meet ethical criteria, both when 
deployed, and considering their evolution after 
deployment and interaction with other products. 


Further Resources 


e Tutt, A. “An FDA for Algorithms.” 
Administrative Law Review (2017): 83-123. 


* Scherer, M. U. “Regulating Artificial 
Intelligence Systems: Risks, Challenges, 
Competencies, and Strategies.” Harvard 
Journal of Law and Technology 29, no. 2 
(2016): 354—400. 











e Desai, D. R., and J. A. Kroll. “Trust But 
Verify: A Guide to Algorithms and the Law." 
Harvard Journal of Law and Technology 
(2018 forthcoming). 








Issue: 
Use of black-box components. 


Background 


Software developers regularly use “black-box” 
components in their software, the functioning 
of which they often do not fully understand. 
“Deep” machine learning processes, which are 
driving many advancements in autonomous 
systems, are a growing source of “black-box” 
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software. At least for the foreseeable future, 

Al developers will likely be unable to build 
systems that are guaranteed to operate exactly 
as intended or hoped for in every possible 
circumstance. Yet, the responsibility for resulting 
errors and harms remains with the humans that 
design, build, test, and employ these systems. 


Candidate Recommendation 


When systems are built that could impact the 
safety or well-being of humans, it is not enough 
to just presume that a system works. Engineers 
must acknowledge and assess the ethical risks 
involved with black-box software and implement 
mitigation strategies. 


Candidate Recommendation 


Technologists should be able to characterize 
what their algorithms or systems are going 

to do via transparent and traceable standards. 

To the degree possible, these characterizations 
should be predictive, but given the nature 

of A/IS, they might need to be more retrospective 
and mitigation oriented. Such standards may 
include preferential adoption of effective design 
methodologies for building “explainable Al” (XAI) 
systems that can provide justifying reasons or 
other reliable “explanatory” data illuminating the 
cognitive processes leading to, and/or salient 
bases for, their conclusions. 


Candidate Recommendation 


Similar to a flight data recorder in the field 

of aviation, this algorithmic traceability can 
provide insights on what computations led to 
specific results that ended up in questionable or 


creative 
commons 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 


dangerous behaviors. Even where such processes 
remain somewhat opaque, technologists should 
seek indirect means of validating results and 
detecting harms. 


Candidate Recommendation 


Software engineers should employ “black-box” 
(opaque) software services or components 
only with extraordinary caution and ethical care, 
as they tend to produce results that cannot be 
fully inspected, validated, or justified by ordinary 
means, and thus increase the risk of undetected 
or unforeseen errors, biases, and harms. 


Further Resources 


e Pasquale, F. The Black Box Society. 
Cambridge, MA: Harvard University Press, 
2015. 


e In the United States, in addition to similar 
commercial endeavors by Oracle and other 
companies, DARPA (Defense Advanced 
Research Projects Agency) recently funded 
a 5-year research program in explainable 
Al (XAI) methodologies. 





e Ananny, M, and K. Crawford. (2016). 
“Seeing without Knowing: Limitations of 
the Transparency Ideal and Its Application 
to Algorithmic Accountability.” New Media 
& Society, December 13, 2016. 











¢ Another excellent resource on these 
issues can be found in Chava Gourarie’s 
“Investigating the Algorithms That Govern 





Our Lives.” Columbia Journalism Review, 
April 14, 2016. These recommended reads 
come at the end of the article: 
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e "How big data is unfair”: A layperson's e “Certifying and removing disparate 
guide to why big data and algorithms are impact”: The computer scientist's 
inherently biased. guide to locating and fixing bias in 


algorithms computationally, by Suresh 


* “Algorithmic accountability reporting: Venkatasubramanian and colleagues. 


On the investigation of black boxes": 











The primer on reporting on algorithms, e The Curious Journalist's Guide to Data: 
by Nick Diakopoulos, an assistant Jonathan Stray's gentle guide to 
professor at the University of Maryland thinking about data as communication, 
who has written extensively on the much of which applies to reporting 
intersection of journalism and algorithmic on algorithms as well. 


accountability. 
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The concept of intelligence can be difficult to precisely define, and there are many 
proposed definitions. Legg and Hutter (2007) surveyed 70-odd definitions of intelligence, 
pulling out the key features and commonalities between them, and settled on the following: 
“intelligence measures an agent's ability to achieve goals in a wide range of environments.” 





In the context of autonomous and intelligent systems (A/IS), artificial general intelligence 
(AGI) is often used to refer to A/IS that perform comparably to humans on intellectual 
tasks, and artificial superintelligence (ASI or superintelligence) is commonly defined 

as “an intellect that is much smarter than the best human brains in practically every field, 
including scientific creativity, general wisdom and social skills" (Bostrom 2014), passing 
some threshold of generality, well-roundedness, and versatility that present-day Al systems 
do not yet achieve. 


Although today’s state-of-the-art A/IS do not match humans in this capacity (since today's 
systems are only capable of performing well in limited and narrow environments or 
domains), many independent researchers and organizations are working on creating AGI 
systems (including leading Al labs like DeepMind, OpenAl, Microsoft, and Facebook's FAIR), 
and most Al experts expect A/IS to surpass human-level intelligence sometime this century 
(Grace et al. 2017). 











When reasoning about the impacts that AGI systems will have, it is tempting to 
anthropomorphize, assume that these systems will have a “mind” similar to that of 

a human, and conflate intelligence with consciousness. Although it should be possible 

to build AGI systems that imitate the human brain, the human brain represents one point 
in a vast space of possible minds (Yampolskiy 2015). AGI systems will not be subject to 
the same constraints and engineering trade-offs as the human brain (a product of natural 
selection). Thus, we should not expect AGI systems to necessarily resemble human 

brains, just as we don't expect planes to resemble birds, even though both are flying 
machines. This also means that familiar faculties of intelligent entities we know like morality, 
compassion, and common sense will not be present by default in these new intelligences. 
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History shows that the largest drivers of change in human welfare, for better and for worse, 
have been developments in science, technology, and economics. Humanity’s ability to 
drive this change is largely a function of our intelligence. Thus, one can think about 
building AGI as automating scientific, technological, and economic innovation. Given the 
disproportionate impact our intelligence has enabled our species to have on the planet 
and our way of life, we should expect AGI systems to have a disproportionate impact on 
our future, on a scale not seen since the Industrial Revolution. As such, the development 
of AGI systems and improvements of those systems toward superintelligence could bring 
about unprecedented levels of global prosperity. However, it is by no means guaranteed 
that the impact of these systems will be a positive one without a concerted effort by 

the A/IS community and other key stakeholders to align them with our interests. 


As with other powerful technologies, the development and use of A/IS have always 
involved risk, either because of misuse or poor design (as simple examples being an 
assembly line worker being injured by a robotic arm or a guard robot running over a child's 
foot). However, as systems approach and surpass AGI, unanticipated or unintended system 
behavior (due to, e.g., architecture choices, training or goal specification failures, mistakes 
in implementation, or mistaken assumptions) will become increasingly dangerous and 
difficult to correct. It is likely that not all AGI-level A/IS architectures are alignable with 
human interests, and as such, care should be taken to analyze how different architectures 
will perform as they become more capable. In addition to these technical challenges, 
technologists will also confront a progressively more complex set of ethical issues during 
the development and deployment of these technologies. 





In section 1 which focuses on technical issues, we recommend that A/IS teams working 
to develop these systems cultivate a “safety mindset,” in the conduct of research in 
order to identify and preempt unintended and unanticipated behaviors in their systems, 
and work to develop systems which are “safe by design.” Furthermore, we recommend 
that institutions set up review boards as a resource to researchers and developers, and 
to evaluate relevant projects and their progress. In Section 2 which focuses on general 
principles, we recommend that the A/IS community encourage and promote the sharing 
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of safety-related research and tools, and that all those involved in the development and 
deployment take on the norm that future highly capable transformative A/IS “should 
only be developed in the service of widely shared ethical ideals, and for the benefit 
of all humanity rather than one state or organization.” (Future of Life Institute 2017) 





Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — Technical 





Issue: 


As A/IS become more capable, 
as measured by the ability to 
perform with greater autonomy 
across a wider variety of 
domains, unanticipated or 
unintended behavior becomes 
increasingly dangerous. 


Background 


A/IS with an incorrectly or imprecisely specified 
objective function (or goals) could behave in 
undesirable ways (Amodei et al. 2016, Bostrom 
2014, Yudkowsky 2008). In their paper, Concrete 
Problems in Al Safety, Amodei et al. describe 
some possible failure modes, including: scenarios 
where the system has incentives to attempt to 
gain control over its reward channel, scenarios 
where the learning process fails to be robust 

to distributional shift, and scenarios where the 
system engages in unsafe exploration (in the 
reinforcement learning sense). Further, Bostrom 
(2012) and Omohundro (2008) have argued 
that AGI systems are likely by default to adopt 
“convergent instrumental subgoals” such as 
resource-acquisition and self-preservation, unless 
the system is designed to explicitly disincentivize 
these strategies. These types of problems are 
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likely to be more severe in systems that are 
more capable (as follows from their increased 
optimization power and broader action space 
range) unless action is taken to prevent them 
from arising. 


In order to foster safety and controllability, 

A/IS that are intended to have their capabilities 
improved to the point where the above issues 
begin to apply should be designed to avoid those 
issues preemptively. When considering problems 
such as these, teams should cultivate a “safety 
mindset” (as described by Schneier [2008] in the 
context of computer security — to anticipate and 
preempt adversaries at every level of design and 
implementation), and suggest that many of these 
problems can likely be better understood by 
studying adversarial examples (as discussed 

by Christiano [2016]) and other A/IS robustness 
and safety research threads. 





Teams working on such advanced levels of A/IS 
should pursue the following goals, all of which 
seem likely to help avert the above problems: 


1. Contribute to research on concrete problems 
in Al safety, such as those described by 
Amodei et al. in Concrete Problems in Al 
Safety, Taylor et al. in Alignment for Advanced 
Machine Learning Systems, and Russell 
et al. in Research Priorities for Robust and 
Beneficial Artificial Intelligence. See also 
the work of Hadfield-Menell et al. (2016) 
and the references therein. 
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Work to ensure that A/IS are transparent, 
i.e., that their internal reasoning processes 
can be understood by human operators. 
This likely involves both theoretical and 
practical research. In particular, teams 
should develop, share, and contribute to 
transparency and debugging tools that make 
the behavior of advanced A/IS easier to 
understand and work with; and teams should 
perform the necessary theoretical research 
to understand how and why a system 

works at least well enough to ensure that 
the system will avoid the above failure 
modes (even in the face of rapid capability 
gain and/or a dramatic change in context, 
such as when moving from a small testing 
environment to a large world). 


Work to build safe and secure infrastructure 
and environments for development, testing, 
and deployment of powerful A/IS. This work 
will provide some protection against risks 
including subversion by malicious external 
attackers, and unsafe behavior arising from 
exploratory learning algorithms. In particular, 
teams should develop, share, and contribute 
to Al safety test environments and tools and 
techniques for “boxing” A/IS (see Babcock 
et al. [2016] and Yampolskiy [2012] for 
preliminary work). 


Work to ensure that A/IS “fail gracefully” 
(e.g., shutdown safely or go into some other 
known-safe mode) in the face of adversarial 
inputs, out-of-distribution errors (see Siddiqui 
et al. [2016] for an example), unexpected 
rapid capability gain, and other large context 
changes. 


Ensure that A/IS are corrigible in the sense 
of Soares et al. (2015), i.e., that the systems 
are amenable to shutdown and modification 
by the operators, e.g., as with Hadfield- 
Menell (2017) and Russell et al. (2016), and 
assist (or at least do not resist) the operators 
in shutting down and modifying the system 
(if such a task is non-trivial). See also the 
work of Armstrong and Orseau (2016). 


Explore methods for making A/IS capable 
of learning complex behaviors and goals 
from human feedback and examples, 

in spite of the fact that this feedback is 
expensive and sometimes inconsistent, e.g., 
as newer variants of inverse reinforcement 
learning attempt. See Evans et al. (2015) 
and Hadfield-Menell et al. (2016). 


Build extensive knowledge layers and 
automated reasoning into systems to expand 
their contextual awareness and common 
sense so undesirable side effects can 

be determined and averted dynamically. 


Candidate Recommendations 


1. 


Teams working on developing AGI systems 
should be aware that many technical 
robustness and safety issues are even 
present in today’s systems and that, given 
more research, some corrective techniques 
for those can likely scale with more complex 
problem manifestations. 


Teams working on developing AGI systems 
should be prepared to put significantly 
more effort into Al safety research as 
capabilities grow. 
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3. Teams working on developing AGI systems 
should cultivate a “safety mindset” like a 
“security mindset,” vigilant of ways they can 
cause harm and invest in preventing those. 





Issue: 


Designing for safety may 

be much more difficult later 
in the design lifecycle rather 
than earlier. 


Background 


Different types of AGI systems are likely to vary 
widely in how difficult they are to align with 

the interests of their operators. As an example, 
consider the case of natural selection, which 
developed an intelligent “artifact” (brains) by a 
process analogous to a simple hill-climbing search 
algorithm. Brains are quite difficult to understand, 
and modifying a brain to be trustworthy when 
given large amounts of resources and unchecked 
power would be extremely difficult or impossible. 


Similarly, systems developed using search/ 
optimization, especially those using multiple 
layers of representations, might be difficult to 
modify/align. At the other end of the spectrum, 
we can imagine systems with more principled 
or explicit designs that are perfectly rational, 
understandable, and easy to modify/align. On 
this spectrum, a system like AlphaGo would be 
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closer to the search/optimization/meta end of 
the spectrum, and Deep Blue closer to the other. 


Realistic AGI systems are likely to fall somewhere 
in between, and will be built by a combination 
of human design and search/optimization 

(e.g., gradient descent, trial-and-error, etc.). 
Developing AGI systems without these concerns 
in mind could result in complicated systems 

that are difficult or impossible to align with the 
interests of its operators, leading to systems 

that are more vulnerable to the concerns 

raised above. 


A relevant analogy for this issue is the 
development of the C programming language, 
which settled on the use of null-terminated 
strings instead of length-prefixed strings for 
reasons of memory efficiency and code elegance, 
thereby making the C language vulnerable to 
buffer overflow attacks, which are to this day one 
of the most common and damaging types of 
software vulnerability. If the developers of C had 
been considering computer security (in addition 
to memory efficiency and code elegance), this 
long-lasting vulnerability could perhaps have 
been avoided. Paying the upfront cost in this case 
would have prevented much larger costs that we 
are still paying today. (It does require skill though 
to envision the types of downstream costs that 
can result from upstream architectural changes.) 


Given that some A/IS development methodologies 
will result in AGI systems that are much easier 

to align with intentions than other methodologies, 
and given that it may be quite difficult to switch 
development methodologies and architectures 
late in the development of a highly capable A/S, 
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great care should be taken by teams developing 
systems intended to eventually reach AGI level 

to ensure that their development methodology, 
techniques, and architecture will result in a system 
that can be easily aligned. (See also the discussion 
of transparency tools above.) 


As a heuristic, when teams develop potentially 
dangerous systems, those systems should be 
“safe by design,” in the sense that if everything 
goes according to plan, then the safety 
precautions discussed above should not be 
necessary (see Christiano [2015] for a discussion 
of a related concept he terms “scalable Al 
control”). For example, a system that has strong 
incentives to manipulate its operators, but which 
cannot do so due to restrictions on the system's 


action space, is not safe by design. Of course, 

all appropriate safety precautions should be used, 
but safeties such as “boxes,” tripwires, monitors, 
action limitations, and so on should be treated 

as fail-safes rather than as a first line of defense. 


Candidate Recommendation 


When designing an advanced A/S, researchers 
and developers should pay the upfront costs to 
ensure, to the extent possible, that their systems 
are “safe-by-design,” and only use external 
restrictions on the system as fail-safes rather than 
as a first line of defense. This involves designing 
architectures using known-safe and more-safe 
technical paradigms as early in the lifecycle 

as possible. 
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Section 2 — General Principles 





Issue: 


Researchers and developers will 
confront a progressively more 
complex set of ethical and 
technical safety issues in the 
development and deployment 
of increasingly capable A/IS. 


Background 


Issues A/IS researchers and developers will 
encounter include challenges in determining 
whether a system will cause unintended 

and unanticipated harms — to themselves, the 
system's users, and the general public — as well 
as complex moral and ethical considerations, 
including even the moral weight of certain 

A/IS themselves or simulations they may produce 
(Sandberg 2014). Moreover, researchers and 
developers may be subject to cognitive biases 
that lead them to have an optimistic view of the 
benefits, dangers, and ethical concerns involved 
in their research. 


Across a wide range of research areas in science, 
medicine, and social science, review boards 
have served as a valuable tool in enabling 

those with relevant expertise to scrutinize 

the ethical implications and potential risks of 
research activities. While A/IS researchers and 
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developers themselves should be alert to such 
considerations, review boards can provide 
valuable additional oversight by fielding a diversity 
of disciplines and deliberating without direct 
investment in the advancement of research goals. 


Organizations should set up review boards to 
support and oversee researchers working on 
projects that aim to create very capable A/IS. 

Al researchers and developers working on such 
projects should also advocate that these boards 
be set up (see Yampolskiy and Fox [2013] for 

a discussion of review boards for Al projects). 
There is already some precedent for this, such 
as Google DeepMind's ethics board (though not 
much is known publicly about how it functions). 


Review boards should be composed of impartial 
experts with a diversity of relevant knowledge 
and experience. These boards should be 
continually engaged from the inception of the 
relevant project, and events during the course 
of the project that trigger special review should 
be determined ahead of time. These types of 
events could include the system dramatically 
outperforming expectations, performing rapid 
self-improvement, or exhibiting a failure of 
corrigibility. Ideally review boards would adhere 
to some (international) standards or best 
practices developed by the industry/field 

as a whole, perhaps through groups like the 
Partnership on Artificial Intelligence, our IEEE 
Global Initiative, or per the Asilomar Al Principles. 
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Review boards should be complemented by other 
measures to draw upon diverse expertise and 
societal views, such as advisory groups, relevant 
workshops and conferences, public engagement 
processes, and other forums for discussion and 
debate. The incorporation of a wide range of 
viewpoints, commensurate with the breadth 
and scale of potential impact, will support A/IS 
researchers and developers in making optimal 
design decisions without relying solely on the 
oversight of review boards. 


Given the transformative impact AGI systems 
may have on the world, it is essential that 
review boards take into consideration the widest 
possible breadth of safety and ethical issues. 
Furthermore, in light of the difficulty of finding 
satisfactory solutions to moral dilemmas and the 
sheer size of the potential moral hazard that one 
team would face when deploying an AGI-level 
system, technologists should pursue Al designs 
that would bring about beneficial outcomes 
regardless of the moral fortitude of the research 
team. Teams should work to minimize the extent 
to which beneficial outcomes from the system 
hinge on the virtuousness of the operators. 


Candidate Recommendation 


1. Organizations working on sufficiently 
advanced A/IS should set up review boards 
to consider the implications of risk-bearing 
proposed experiments and development. 


2. Technologists should work to minimize 
the extent to which beneficial outcomes 
from the system hinge on the virtuousness 
of the operators. 
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Issue: 


Future A/IS may have the 
capacity to impact the world 
on a scale not seen since the 
Industrial Revolution. 


Background 


The development of very capable A/IS could 
completely transform not only the economy, 

but the global political landscape. Future A/IS 
could bring about unprecedented levels of 

global prosperity, health, and overall well-being, 
especially given the potential impact of 
superintelligent systems (in the sense of Bostrom 
[2014]). It is by no means guaranteed that this 
transformation will be a positive one without a 
concerted effort by the A/IS community to shape 
it that way (Bostrom 2014, Yudkowsky 2008). 


The academic A/IS community has an admirable 
tradition of open scientific communication. 
Because A/IS development is increasingly 
taking place in a commercial setting, there 

are incentives for that openness to diminish. 
The A/IS community should work to ensure that 
this tradition of openness be maintained when 
it comes to safety research. A/IS researchers 
and developers should be encouraged to freely 
discuss Al safety solutions and share best 
practices with their peers across institutional, 
industry, and national boundaries. 
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Furthermore, institutions should encourage A/IS 
researchers and developers, who are concerned 
that their lab or team is not following safety best 
practices, to raise this to the attention of the 
wider A/IS community without fear of retribution. 
Any group working to develop capable A/IS 
should understand that, if successful, their 
technology will be considered both extremely 
economically and politically significant. 
Accordingly, for non-safety research and results, 
the case for openness is not quite so clear-cut. 
It is necessary to weigh the potential risks of 
disclosure against the benefits of openness, 

as discussed by Bostrom (2016) and Krakovna 


(2016). 


In his book Superintelligence, philosopher Nick 
Bostrom proposes that we adopt a moral norm 
which he calls the common good principle: 
“Superintelligence should be developed only 
for the benefit of all humanity and in the service 
of widely shared ethical ideals” (Bostrom 2014, 
254). We encourage researchers and developers 
aspiring to develop these systems to take on 
this norm. It is imperative that the pursuit and 
realization of AGI systems be done in the 
service of the equitable, long-term flourishing 

of civilization. 


In 2017, broad coalitions of Al researchers, 
ethicists, engineers, businesspeople, and social 
scientists came together to form and to endorse 
the Asilomar Al Principles (Future of Life Institute 
2017), which includes the relevant principles 
“14) Shared Benefit: Al technologies should 
benefit and empower as many people as 
possible. ... 15) Shared Prosperity: The economic 
prosperity created by Al should be shared 
broadly, to benefit all of humanity. ... 23) Common 
Good: Superintelligence should only be developed 
in the service of widely shared ethical ideals, 

and for the benefit of all humanity rather than 
one state or organization." 





Candidate Recommendations 


1. Adopt the stance that superintelligence 
should be developed only for the benefit 
of all of humanity. 


2. De-stigmatize and remove other soft 
and hard barriers to Al researchers and 
developers working on safety, ethics, 
and beneficence, as well as being open 
regarding that work. 
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Autonomous and Intelligent systems (A/IS) are developing faster than the supporting 
standards and regulation required for transparency and societal protections can keep pace. 
The impact of these systems on society is direct and considerable. 


A/IS require data to fuel learning, and inform automatic decision-making. Increasingly this 
data is personal data, or personally identifiable information, known as PII. PII is defined 

as any data that can be reasonably linked to an individual based on their unique physical, 
digital, or virtual identity. As a result, through every digital transaction (explicit or observed) 
humans are generating a unique digital shadow of their physical self. 


Ethical considerations regarding data are often focused largely on issues of privacy — what 
rights should a person have to keep certain information to themselves or have input into 
how it is shared? However, individuals currently lack clarity around how to access, organize, 
and share their data to ensure unintended consequences are not the Laws are generally 
enforceable result. Without clarity, these issues will continue to reflect negatively on the 
proliferation of the A/IS industry. 


The aim of this Committee is to set out the ethical considerations in the collection and 

use of personal data when designing, developing, and/or deploying A/IS. Furthermore, 

to entreat all global (A/IS) technologists (academics, engineers, programmers, manufacturers, 
and policy makers) to proactively prioritize and include individuals in the data processes 
that directly relate to their identity. 


There is a fundamental need for people to have the right to define access and provide 
informed consent with respect to the use of their personal data (as they do in the physical 
world). Individuals require mechanisms to help curate their unique identity and personal data 
in conjunction with policies and practices that make them explicitly aware of consequences 
resulting from the bundling or resale of their personal information and life experiences. 


Enabling individuals to curate their identities and manage the ethical implications of their 
data use will remain essential to human culture everywhere in the world. While some may 
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choose only minimum compliance to legislation like the European General Data Protection 
Regulation (GDPR), forward-thinking organizations will shift their data strategy (marketing, 
product, and sales) to enable methods of harnessing volunteered intentions from customers 
(or in governmental contexts, citizens), versus only invisibly tracking their attention or actions. 


For individuals to be at the center of their data, policy makers and society at large will need 
to rethink the nature of standards and human rights as they have been applied to the 
physical world and to re-contextualize their application in the digital world. While standards 
exist, or are in production relating to augmented and virtual reality, human rights law, 
privacy and data, it is still largely not understood how human agency, emotion, and the legal 
issues regarding identity will be affected on a large scale by society once A/IS technologies 
become ubiquitous. 


The goal of the analysis of these ethical issues and considerations by this Committee 
regarding data usage and identity is to foster a positive and inclusive vision for our shared 
future. To accomplish this goal, this document is focused on the following themes: 


1. Digital Personas 





2. Regional Jurisdiction 
3. Agency and Control 
4. Transparency and Access 


5. Symmetry and Consent 


We have also created an Appendix document listing key resources referenced in the 
following section. 
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Addressing these issues and establishing safeguards prioritizing the protection and assets 

of individuals regarding privacy and personal data in the realms of A/IS is of paramount 
importance today. To that end, since the creation of the first draft of Ethically Aligned Design 
this Committee recommended ideas for the following IEEE Standards Working Groups which 
have been and approved and are free for all to join (click on links for details): 


e IEEE P7002™, Data Privacy Process 





IEEE P7004™, Standard on Child and Student Data Governance 





* IEEE P7005™, Standard on Employer Data Governance 





* IEEE P7006™, Standard for Personal Data Artificial Intelligence (Al) Agent 





The goal of this Committee is that our recommendations, in conjunction with the 
development and release of these Standards once adopted, will expedite the prioritization 
and inclusion of all global individuals in the data processes that directly relate to their identity. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — Digital Personas 


While many individuals may not currently have 
the ability to claim their identity (in the case of 
refugees, etc.), as a rule society understands how 
to apply the legal concepts of identity in real-life 
situations. In digital or virtual realms, however, 
our personas are fluid — individuals can be 
avatars in gaming situations or take on a different 
tone in various social networking settings. 
Behaviors regarding our personas considered 
normal in real-life are not directly applicable in 
the augmented, virtual and mixed reality worlds 
most individuals will soon be inhabiting on a 
regular basis in the near future. In regards to the 
algorithms powering Al, or the affective sensors 
becoming standard features in autonomous 
vehicles, or companion robots, etc., how A/IS 
affects our digital personas through use or 
misuse of our data is critical to understand, 
monitor, and control. 
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Issue: 


Individuals do not understand 
that their digital personas 

and identity function differently 
than in real life. This is a concern 
when personal data is not 
accessible by an individual and 
the future iterations of their 
personas or identity cannot 

be controlled by them, but by 
the creators of the A/IS they use. 


Background 


A/IS created from personal experiences is 
different from Al created from farming or climate 
data. Society has had traditional safeguards on 
the use and application of personal information 


to encourage innovation and to protect minorities. 


Traditional systems for medicine and law limit 
secrecy and favor regulation of professionals 

at the edges over centralized hierarchical 
corporations. For example, almost 100% of 
intellectual property in the domains of medicine 
and law is open, peer-reviewable, and can be 
taught to anyone, anywhere. 
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However, the emergence of the Internet of 
Things (loT) and augmented reality/virtual reality 
(AR/VR) means personal information forms 

a foundation for every system being designed. 
This data acts as the digital representation and 
proxy for our identity. From birth, the different 
roles individuals take on in life provide specific 
contexts to the data they generate. Previously 
these contexts and roles enabled individuals to 
maintain some level of privacy due to the siloes 
of collection. Now, as the prospect of an omni- 
connected world approaches, those silos are 
being replaced by horizontal integrations that put 
the digital versions of personas and roles at risk. 
It is therefore important that citizens understand 
these roles and their related data to assess 

the downstream (further) consequences of its 
aggregation. Digital personas/roles include: 


* — Pre-birth to post-life digital records 
(health data) 


e — Birth and the right to claim citizenship 
(government data) 


* Enrollment in school (education data) 
e Travel and services (transport data) 


e Cross-border access and visas 
(immigration data) 


e Consumption of goods and services 
(consumer and loyalty data) 


e Connected devices, loT and wearables 
(telecommunications data) 


* Social and news networks (media and 
content data) 


e Professional training, internship, and work 
(tax and employment data) 


e Societal participation (online forums, 
voting and party affiliation data) 


e Contracts, assets, and accidents (insurance 
and legal data) 


e — Financial participation (banking and 
finance data) 


e Death (digital inheritance data) 


By the time individuals reach early adulthood, 
they are simultaneously acting across these roles, 
generating vast amounts of personal data that 

is highly contextual and easy to identify and link 
directly to an individual. If an individual's digital 
shadow is a proxy of their physical self, then 
technologists and policy makers must address 
the transparency, control, and asymmetry of 

how personal data is collected and used to 
enable A/IS. A/IS technologists need to recognize 
the coercive nature of many current identity 
schemes — such as hidden tracking by advertising 
brokers — and adopt privacy-preserving identity 
practices such as same-domain pseudonymous 
identifiers and self-sovereign identity. 


Candidate Recommendation 


The ethics of creating secret and proprietary 
A/IS from people's personally identifiable 
information (PII) need to be considered based 
on the potential impact to the human condition. 
To preserve human dignity, policies, protections, 
and practices must provide all individuals the 
same agency and control over their digital 
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personas and identity they exercise in their real- 
world iterations no matter what A/IS may be in 
place to monitor, assist, or interact with their data. 


Further Resources 
* Blockchain Identity (Rebooting Web-of-Trust). 





* W3C Credentials Community Group. 





e HIE of One. 





How can an individual 
define and organize his/her 
personal data and identity 
in the algorithmic era? 


Background 


Identity is emerging at the forefront of the risks 
and opportunities related to use of personal 

data for A/IS. Across the identity landscape there 
is increasing tension between the requirement 
for federated identities (all data linked to a natural 
and identified natural person) versus a range 

of identities (personas) that are context specific 
and determined by the use-case, for example 
opening a bank account, crossing a border, or 
ordering a product online. New movements, such 
as Self-Sovereign Identity — defined as the right 
of a person to determine his or her own identity 
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— are emerging alongside legal identities 
(issued by governments, banks, and regulatory 
authorities) to help put individuals at the center 
of their data in the algorithmic age. 


Personas (an identity that acts as a proxy) and 
pseudonymity are also critical requirements for 
privacy management since they help individuals 
select an identity that is appropriate for the 
context they are in or wish to join. In these 
settings, trust transactions can still be enabled 
without giving up the “root” identity of the user. 
For example, it is possible to validate a user is 
over 18 (for adult content) or eligible for a service 
(postcode confirmation). Attribute verification 
(comprising the use of empowered persona 
usage by an individual) will play a significant role 
in enabling individuals to select the identity that 
provides access without compromising agency. 
This type of access is especially important in 
dealing with the myriad algorithms interacting 
with data representing tiny representations of our 
identity where individuals typically are not aware 
of the context for how their data will be used. 


Candidate Recommendation 


Individuals should have access to trusted identity 
verification services to validate, prove, and 
support the context-specific use of their identity. 
Regulated industries and sectors such as banking, 
government, and telecommunications should 
provide data-verification services to citizens and 
consumers to provide greatest usage and control 
for individuals. 
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Further Resources 


+ The Inevitable Rise of Self-Sovereign Identity 
by The Sovrin Foundation. 





* See Identity Examples in the Appendix 
Document for this section. 








e IEEE P7006™, Standard for Personal Data 
Artificial Intelligence (Al) Agent Working 
Group. This Standards Working Group 
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is free and open to anyone wishing to join 
and addresses issues relating to how an 
individual could have the ubiquitous and 
always-on services of a personalized Al agent 
to ensure their identity is protected and has 
symmetry with the A/IS their data comes 
into contact with at all times. 
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Section 2 — Regional Jurisdiction 


Legislation regarding personal data varies 
widely around the world. Beyond issues of data 
operability issues when transferring between 
country jurisdictions, rights of individuals and 
their access and usage of data depends on 

the regions and laws where they live. Much 

of A/IS ethics involves the need to understand 
cultural aspects of the systems and services an 
organization wishes to create for specific users. 
This same attention must be given to how data 
related to A/IS are positioned from a regional 
perspective to best honor the use, or potential 
abuse of the global citizens’ data. A/IS will also 
be subject to regional regulation, for example 
under the General Data Protection Regulation 
(GDPR), European citizens may have specific 


rights of redress where Al or AS has been used. 





Issue: 

Country-wide, regional, 

or local legislation may 
contradict an individual's 
values or access and control 
of their personal data. 


@greative 
commons 


Background 


Ethical considerations regarding data are often 
focused largely on issues of privacy — what 
rights should a person have to keep certain 
information to themselves, or have input into 
how it is shared? While rhetoric in various circles 
stating, “privacy is dead” may be someone's 
personal opinion reflecting their values, privacy 
is nonetheless a fundamental human right 
recognized in the UN Declaration of Human 
Rights, the International Covenant on Civil and 
Political Rights, and in many other international 
and regional treaties. 





However, this fundamental right is not universally 
recognized or supported. It is also culturally 
contextual and nuanced. It is therefore critical 

to understand the jurisdictional and specific legal 
requirements that govern the access and use 

of personal information when developing A/IS 
solutions. These include, but are not limited to: 


e Europe; the introduction of the General 
Data Protection Regulation (GDPR), Personal 
Services Directive Il (PSD2), and ePrivacy. 
These new regulations carry substantial 
fines for non-compliance. Depending on the 
nature and circumstances of the violation, 
these penalties may include: 





e A warning in writing in cases of first 
and non-intentional non-compliance 


e Regular periodic data protection audits 
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e Afine up to 10,000,000 EUR or up 
to 2% of the annual worldwide turnover 
of the preceding financial year in case 
of an enterprise, whichever is greater 
(Article 83, Paragraph 4) 





e A fine up to 20,000,000 EUR or up 
to 4% of the annual worldwide turnover 
of the preceding financial year in case 
of an enterprise, whichever is greater 
(Article 83, Paragraph 5 and 6) 





United States: The United States lacks 

a single “baseline” privacy regime; instead, 
policies and procedures affecting the 
collection and use of PII varies based 

on type of information and which entity 
possesses the data. Laws, for example, afford 
certain procedural requirements around 
financial data, certain protected health 
information, and children’s data. Laws are 
generally enforceable by state and federal 
regulators (including the Federal Trade 
Commission and state attorney general), 
though individuals may have private rights 
of action under state law or certain federal 
laws such as the Video Privacy Protection 
Act, which governs disclosures of identifiable 
video rental records, and the Fair Credit 
Reporting Act, which provides access and 
rights to consumer reports used for eligibility 
determinations. See also: Jurisdiction 
Examples in the Appendix Document for 
this section. 





e Australia: In addition to strict privacy 
regulation, the Australian Productivity 
Commission issued reports in 2016 and 
2017 acknowledging that personal 
information is a personal asset and therefore 
recognized the need for Australians to have 
control with respect to its collection and 
use. At the time of publication, The Australian 
Federal Government is in the process of 
using these reports to inform the drafting 
of new personal data regulation. 


e Japan: The Act on the Protection of Personal 
Information was amended in 2016. The act 
precisely defines the definition of personal 
information; however, the concept of privacy 
is not explicitly stated. In this sense, the act 
is deemed as a practice-oriented law. The 
new concept of anonymously processed 
information is introduced which is produced 
to make it impossible to identify a specific 
individual. In addition, it can be transferred 
to, and used by, the third parties without 
the data subject's consent. The method 
of producing anonymously processed 
information will be determined on a sector- 
by-sector basis because each sector has 
distinct constraints and purposes of personal 
information. 


Additionally, there is growing evidence that 

not providing clear consent (regarding personal 
data usage) decreases mental and emotional 
well-being. The rapid rise in ad blocking tools 





or lowering of consumer trust via reports 

of non-ethically driven online studies provides 
tangible evidence toward the failure of these 
clandestine efforts. 
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Candidate Recommendation Further Resources 
While specific uses of data must be taken in * Amended Act on the Protection of Personal 
context of the regions where specific legislation Information in Japan. 


applies, individuals should always be provided 
access to, and control of, their data to ensure 
their fundamental human rights are honored 
without fear of the risk of breaking applicable laws. 


e Outline of the Amended Personal Information 
Protection Act in Japan. 
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Section 3 — Agency and Control 


Agency is the capacity of individuals to act 
independently and to exercise free choice, a 
quality fundamental to democratic ideals. Central 
to human agency is control. As society moves 
towards complete connectivity, humans will 
require tools and mechanisms to enable agency 
and control over how their personal data is 
collected and used. When people do not have 
agency over their identities political participation 
is impossible, and without political participation 
ethics will be decided by others. As the rise 

of algorithms accessing people's data relating 

to their identities continues, there is increased 
risk of loss of agency and well-being, adding 

the potential for depression and confusion along 
with the lack of clear ways to contribute ideas 

in an open and democratic fashion. 





Issue: 


To understand the role of 
agency and control within A/IS, 
it is critical to have a definition 
and scope of personally 
identifiable information (PII). 
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Background 


Different laws and regulations around the globe 
define the scope of PII differently. The use of 
data analytics to derive new inferences and 
insights into both personal data and technical 
metadata raises new questions about what types 
of information should be considered PII. This 

is further complicated by machine learning and 
autonomous systems that access and process 
data faster than ever before. 


Multiple global bodies believe PII is a sovereign 
asset belonging to an identified individual. PII, 
or personal data, is defined as any data that can 
be reasonably linked to an individual based on 
their unique physical, digital, or virtual identity. 
PII protections are often related to the U.S. 
Fourth Amendment, as the right of the people 
to be secure in their persons, houses, papers, 
and effects. 


As further clarification, the European Union 
definition of personal data set forth in the 

Data Protection Directive 95/46/ECI vi, defines 
personal data as “any information relating to 

an identified or identifiable natural person.” 
Identifiable when? The question asked today 
will have a very different answer tomorrow given 
that all A/IS person-level or device-level data 

is identifiable if the tech advances and the data 
is still available. Agency requires that the control 
be exercised by the subject at the time the data 
is used, not at the time the data is collected. 
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Overall, personal data reflects self-determination 
and the inalienable right for an individual to be 
able to access and control the attributes of their 
physical, digital, and virtual identity. 


Candidate Recommendation 


Individuals should have access to means that 
allow them to exercise control over use of 
personal data at the time the data is used. 

If that agency and control is not available, 
person-level data needs to either be aggregated 
into larger cohorts and the person-level data 
deleted. PII should be defined as the sovereign 
asset of the individual to be legally protected 
and prioritized universally in global, local, and 
digital implementations regardless of whether 
deemed to be de-identified in the way it 

is stored. 


Further Resources 


* Determining What Is Personal Data, 
U.K. Information Commissioner's Office. 








* Electronic Communications Privacy Act. 





* Open PDS. 


e IEEE Digital Inclusion through Trust and 
Agency Industry Connection Program. 





e HIE of One — a patient-owned and controlled 
standards-based, open source EHR, so 
patients can collect, aggregate, and share 
their own data. 
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Issue: 


What is the definition of 
control regarding personal 
data, and how can it be 
meaningfully expressed? 


Background 

Most individuals believe controlling their personal 
data only happens on the sites or social networks 
to which they belong, and have no idea of the 
consequences of how that data may be used 

by others tomorrow. Providing individuals with 
tools, like a personal data cloud, can empower 
users to understand how their data is an asset 
as well as how much data they produce. Tools 
like personal data vaults or clouds also let 
individuals organize their data around various 
uses (medical, social, banking). Control enables 
individuals to also assert a version of their own 
terms and conditions. 


In the current context of A/IS technologies, and 
in the complex and multi-level or secondary 
uses of data, it is important to be clear about the 
boundaries of control for use of personal data 
that can affect an individual directly compared 
to collection of data for aggregated or systematic 
work (and exceptions for approved research). 
For example, an individual subway user's travel 
card, tracking their individual movements, should 
be protected from uses that identify or profile 
that individual to make inferences about his/her 
likes or location generally, but could be included 
in the overall travel systems management to 
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aggregate user data into patterns for scheduling 
and maintenance as long as the individual-level 
data is deleted. 


The MyData movement combines related 
initiatives, such as Self Data, Vendor Relationship 





Management, Internet of Me, and Personal 
Information Management Systems (PIMS) under 
a common cause to empower individuals with 
their personal data. The Declaration of MyData 
Principles highlights human-centric control 

of personal data as one of core principles, 
emphasizing that people should be provided 
with the practical means to understand and 
effectively control who has access to data about 
them and how it is used and shared. In detail, 
the MyData Declaration states: "We want privacy, 
data security and data minimization to become 
standard practice in the design of applications. 
We want organizations to enable individuals to 
understand privacy policies and how to activate 
them. We want individuals to be empowered to 
give, deny or revoke their consent to share data 
based on a clear understanding of why, how and 
for how long their data will be used. Ultimately, 
we want the terms and conditions for using 
personal data to become negotiable in a fair 
way between individuals and organizations.” 











Candidate Recommendation 


Personal data access and consent should be 
managed by the individual using systems that 
provide notification and an opportunity for 
consent at the time the data is used, versus 
outside actors being able to access personal data 
outside of an individual's awareness or control. 


Further Resources 
e — Project VRM — vendor relationship 
management (VRM) tools and frameworks. 


e Kuan Hon, W. K., C. Millard, and I. 
Walden. “The Problem of ‘Personal Data’ 





in Cloud Computing — What Information 
Is Regulated? Cloud of Unknowing, Part 
1." Queen Mary School of Law Legal 
Studies Research Paper No. 75/2011; 
International Data Privacy Law 1, no. 4 
(2011): 211-228. 








* Boyd, E. B. “Personal.com Creates an 
Online Vault to Manage All Your Data.” 
Fast Company, May 7, 2012._ 








* Meeco Life Management Platform. Personal 
cloud, attribute wallet and personal data 
management tools, consent engine and 
dual sided permission APIs. 





e MyData2017. Declaration of MyData 
Principles. 


e  Poikola, A. K. Kuikkaniemi, and H. Honko 
(Ministry of Transport and Communications). 
MyData — A Nordic Model for Human- 
Centered Personal Data Management 
and Processing. Finland: Prime Minister's 
Office, 2014. 











e Hasselbalch, G., and P. Tranberg. “Personal 
Data Stores” (chapter 12), in Data 
Ethics: The New Competitive Advantage. 
Publishare, 2016. 


* GDPR Article 20, Right to Data Portability, 





Article 29 Working Party, Brussels, 2016. 
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e Thurston, B. “A Radical Proposal for 
Putting People in Charge of Their Data.” 
Fast Company, May 11, 2015. 








e de Montjoye, Y.-A., Wang, S. S., and Pentland, 
A. S. “openPDS: Protecting the Privacy of 
Metadata through SafeAnswers.” PLoS ONE 
9, no. 7 (2014): e98790. 








* Definition of the right to be forgotten. 





e — IEEE Digital Inclusion through Trust and 
Agency. The Industry Connection Program 
develops comprehensive roadmaps, industry 
action reports, and educational platforms 
working to address issues around cyber- 
identity, digital personas, distributed ledger 
technology, and inclusion of underserved and 
vulnerable. 





* See “The Attribute Economy 2.0,” a multi- 
authored paper published by Meeco. 





* The Path to Self-Sovereign Identity. 
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uPort is an open source software project 
to establish a global, unified, sovereign 
identity system for people, businesses, 
organizations, devices, and bots. The 
Ethereum based self-sovereign identity 
system now in alpha testing. 





Sovrin—identity for all. The Sovrin Foundation 
describes self-sovereign identity (SSI) as 
“an identity that is 100% owned and 
controlled by an individual or organization. 
No one else can read it, use it, turn it 

off, or take it away without its owner's 
explicit consent.” 





Nichol, P. B. “A Look at India’s Biometric ID 
System: Digital APIs for a Connected World.” 
CIO Perspectives, February 23, 2017. 








See also Appendix 3: Digital Divide and 
Pay for Privacy. 





See also Appendix 4: Examples of Agency 
and Transparency. 





See also Appendix 5: Can Personal Data 
Remain Anonymous? 
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Section 4 — Transparency and Access 


Much of the contention associated with the 
concept of “privacy” actually relates to access. 
Challenges often arise around transparency 

and providing an explicit understanding of the 
consequences of agreeing to the use of people's 
personal data. This is complicated by the 
data-handling processes behind true “consent.” 
Privacy rights are often not respected in the 
design and business model of services using 
said data. They obscure disclosure of the ways 
the data is used and make it hard to know what 
data was used. This can be especially evident 
via the invisible algorithms representing multiple 
services that access people's data long after 
they've provided original access to a service 

or their partners. 


If individuals cannot access their personal data 
and account for how it is used, they cannot 
benefit from the insights that the data could 
provide. Barriers to access would also mean 
that individuals would not be able to correct 
erroneous information or provide the most 
relevant information regarding their lives to 
trusted actors. Transparency is also about 
notification. It is important that an individual 

is notified when their data is collected, and 
what usage is intended. In accordance with 
the GDPR, consent must be informed, explicit, 
and unambiguous. 
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Issue: 


It is often difficult for users 

to determine what information 
a service provider or A/IS 
application collects about them 
at the time of such aggregation/ 
collection (at the time of 
installation, during usage, 

even when not in use, after 
deletion). It is difficult for users 
to correct, amend, or manage 
this information. 


Candidate Recommendation 


Service providers should ensure that personal 
data management tools are easy to find and 
use within their service interface. Specifically: 


* The data management tools should make 
it clear who has access to a user's data and 
for what purpose, and (where relevant) allow 
the user to manage access permissions. 


e There should be legal, reputational, and 
financial consequences for failing to adhere 
to consent terms. 


e It should be easy for users to remove their 
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data from the service. (Note: This is a 
GDPR requirement. It may not be mandated 
in the United States or for other services in 
countries outside of the EU, but represents 
a best-in-class practice to follow.) 
Organizations should create open APIs to 
their data services so that customers can 
access their data and governments should 
share the data they collect about their 

users directly with individuals and encourage 
them to ensure its accuracy for mutual 
value to combat the rising issue of dirty data. 


Further Resources 


e The User Managed Access Standard, 
proposed by The Kantara Initiative, provides 
a useful model to address these types 
of use cases. 


* Surveys about how adults feel about health 
IT in 2005 and 2016 show that distrust of 
health technology has grown from 13% that 
withheld data from providers due to mistrust 
to 89%. 

















Issue: 


How do we create privacy 
impact assessments related 
to A/IS? 


Background 


Because the ethical implications of intelligent 
systems are so difficult to discern, interested 
parties would benefit from analytical tools to 
implement standards and guidelines related to 
A/IS and privacy impacts. Like an environmental 
impact study or the GDPR privacy impact 
assessments, A/IS impact assessments would 
provide organizations with tools to certify their 
products and services are safe and consistent 
for the general public. 


Candidate Recommendation 


A system to assess privacy impacts related to A/IS 
needs to be developed, along with best practice 
recommendations, especially as automated 
decision systems spread into industries that are 
not traditionally data-rich. 


Further Resources 


In the GDPR in the EU, there is a requirement 
for a privacy impact assessment. The full report 
created by PIAF, The Privacy Impact Assessment 
Framework can be found here. In the report, 

of interest is Section 10.3, “Best Elements” 
whose specific recommendations provide 
insights into what could be emulated to 

create an Al impact assessment, including: 








e PIA guidance documents should be aimed 
at not only government agencies but also 
companies or any organization initiating 
or intending to change a project, product, 
service, program, policy, or other initiative 
that could have impacts on privacy. 
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e PIAs should be undertaken about any project, 
product, service, program, or other initiative, 
including legislation and policy, which are 
explicitly referenced in the Victoria Guide 
and the UK Information Commissioner's 
Office (ICO) Handbook. 


Information privacy is only one type of privacy. 
A PIA should also address other types of 
privacy, e.g., of the person, of personal behavior, 
of personal communications, and of location. 


e PIAF Consortium. "PIAF: A Privacy Impact 
Assessment Framework for Data Protection 
and Privacy Rights,” 2011. Section 10.3. 











* See the Personalized Privacy Assistant 
for a project applying these principles. 





e While not explicitly focused on PIAs 
or Al, IEEE P7002™ Data Privacy Process 
is a Standards Working Group still open 
to join focused on these larger issues of 
data protection required by the enterprise 
for individuals’ data usage. 





e Usable Privacy Policy project for examples 
of how difficult privacy policies can be 
to maneuver. 





* See also Appendix 4: Examples of Agency 
and Transparency. 
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Issue: 


How can Al interact with 
government authorities to 
facilitate law enforcement and 
intelligence collection while 
respecting rule of law and 
transparency for users? 


Background 


Government mass surveillance has been 

a major issue since allegations of collaboration 
between technology firms and signals 
intelligence agencies such as the U.S. National 
Security Agency and the U.K. Government 
Communications Headquarters were revealed. 
Further attempts to acquire personal data by law 
enforcement agencies, such as the U.S. Federal 
Bureau of Investigation, have disturbed settled 
legal principles regarding search and seizure. 

A major source of the problem concerns the 
current framework of data collection and storage, 
which puts corporate organizations in custody of 
personal data and detached from the generators 
of that information. Further complicating this 
concern is the legitimate interest that security 
services have in trying to deter and defeat 
criminal and national security threats. 





Candidate Recommendations 


Personal privacy A/IS tools such as IEEE P7006™ 
have the potential to change the data paradigm 
and put the generators of personal information 
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at the center of collection. This would re-define 


the security services’ investigative methods to 


pre-Internet approaches wherein individuals 


would be able to control their information while 


providing custody to corporate entities under 
defined and transparent policies. 


Such a construct would mirror pre-Internet 
methods of information management in which 
individuals would deposit information in narrow 
circumstances such as banking, healthcare, 

or in transactions. This personal data Al agent 





would include root-level settings that would 


automatically provide data to authorities after 


they have satisfied sufficiently specific warrants, 


subpoenas, or other court-issued orders, unless 
authority has been vested in other agencies by 
local or national law. Further, since corporately 
held information would be used under the 
negotiated terms that the A/IS agent facilitates, 
authorities would not have access unless legal 
exceptions were satisfied. This would force 
authorities to avoid mass collection in favor 

of particularized efforts: 


e The roots of the personal privacy A/IS should 
be devoid of backdoors that allow intrusion 
under methods outside of transparent legal 
authority. Otherwise, a personal A/IS could 
feed information to a government authority 
without proper privacy protection. 


* Nuanced technical and legal techniques 
to extract warranted information while 
segregating and avoiding other information 
will be crucial to prevent overreach. 


Each request for data acquisition must come 
on a case-by-case basis versus an ongoing 
access form of access, unless the ongoing 
access has become law. 


Data-acquisition practices need to factor 

in the potential status of purely virtual 
representations of a citizen's identity, whether 
they do not have formal country of origin 
(physical) status, or their virtual identity 
represents a legal form of identity. 


Phasing in personal privacy Als will mitigate 
risks while pre-empting reactive and 
disruptive legislation. 


Legal jurisdiction over personal privacy 
A/IS access will need to be clarified. 


Further Resources 


UNECE. “Evaluating the Potential of 
Differential Privacy Mechanisms for Census 
Data.” Work Session on Statistical Data 
Confidentiality 2013. Ottawa, October 28, 
2013. 








CASD — Le Centre D'Accès Sécurisé Aux 
Données (The Secure Data Access Centre) 
is equipment that allows users, researchers, 
data scientists, and consultants to access 
and work with individual and highly detailed 
microdata, which are therefore subject 

to confidentiality measures, in the most 
secure conditions. 








Initiatives such as OPAL (for Open 
Algorithms), a collaborative project being 
developed by a group of partners committed 
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to leveraging the power of platforms, big 
data, and advanced analytics for the public 
good in a privacy-preserving, commercially 
sensible, stable, scalable, and sustainable 
manner. 


¢ Ohm, P. “Sensitive Information.” Southern 
California Law Review 88 (2015): 
1125-1196. 


e Y-A. de Montjoye, L. Radaelli, V. K. Singh, 
A. S. Pentland. “Unique in the Shopping- 
Mall: On the Reidentifiability of Credit Card 








Metadata.” Science 347 (2015): 536-539. 


* Sanchez, D., S. Martinez., and J. Domingo- 
Ferrer. “Comment on ‘Unique in the 
Shopping Mall: On the Reidentifiability 
of Credit Card Metadata’.” Science 351, 
no. 6279 (2016): 1274-1274. 
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Polonetsky, J., and O. Tene. “Shades of 
Gray: Seeing the Full Spectrum of Practical 
De-ldentification.” Santa Clara Law Review 
56, no. 3 (2016): 593-629. 





Narayanan, A., and V. Shmatikov, “Robust 
De-anonymization of Large Datasets 
(How to Break Anonymity of the Netflix 


Prize Dataset).” February 5, 2008. 


de Montjoye, Y.-A., C. A. Hidalgo, M. 
Verleysen, and V. D. Blondel. “Unique in 
the Crowd: The Privacy Bounds of Human 
Mobility.” Scientific Reports 3, no. 1376 
(2013). doi: 10.1038/srep01376 











Coyne, A. “Government Pulls Dataset That 
Jeopardised 96,000 Employees.” iTnews, 
October 6, 2016. 








Cowan, P. “Health Pulls Medicare Dataset 
After Breach of Doctor Details.” iTnews, 
September 29, 2016. 
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Section 5 — Symmetry and Consent 


Widespread data collection followed by the 
emergence of A/IS and other automated/ 
autonomous data processing has placed 
tremendous strain on existing conceptions 

of “informed consent.” This has created a vast 
asymmetry between the volume of organizations 
tracking individuals versus the tools allowing 
those individuals to fully understand and 
respond to all these tracking signals. 


Legal frameworks such as the GDPR rely on the 
notion that data subjects must provide “freely 
given, specific, informed, and unambiguous” 
consent to certain data processing. Heavy 
reliance on a system of “notice and choice” 
has shifted the burden of data protection away 
from data processors and onto individual data 
subjects. A/IS can exacerbate this trend by 
complicating risk assessments of data sharing. 
When A/IS data transfer is done incorrectly 

it may alter or eliminate user interfaces, limiting 
choice and consent. 


A/IS presents a new opportunity to offer 
individuals/end users a “real choice” with respect 
to how information concerning them is collected, 
used, and shared. Researchers are working 

to solve this issue in some contexts, but design 
standards and business incentives have yet 

to emerge. 
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Issue: 


Could a person have a 
personalized privacy Al or 
algorithmic agent or guardian? 


Background 


For individuals to achieve and retain parity 
regarding their personal information in the 
algorithmic age, it will be necessary to include 

a proactive algorithmic tool that acts as their 
agent or guardian in the digital, and “real” world. 
("Real” meaning a physical or public space where 
the user is not aware of being under surveillance 
by facial recognition, biometric, or other tools 
that could track, store, and utilize their data 
without pre-established consent or permission). 
The creation of personalized privacy A/IS would 
provide a massive opportunity for innovation 

in A/IS and corporate communities. There is 
natural concern that the rights of the individual 
are protected in the face of such opportunities. 


The sophistication of data-sharing methodologies 
has evolved so these scenarios could evolve 
from an “either/or” relationship: “We get all of 
your data for this project, or you provide nothing 
and hinder this work") to a “Yes and” relationship 
— by allowing individuals to set their preferences 
for sharing and storing their data. An additional 
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benefit of finer-grained control of consent 

is that individuals are more likely to trust the 
organizations conducting research and provide 
more access to their data. 


The guardian could serve as an educator and 
negotiator on behalf of its user by suggesting 
how requested data could be combined with 
other data that has already been provided, inform 
the user if data is being used in a way that was 
not authorized, or make recommendations to the 
user based on a personal profile. As a negotiator, 
the guardian could negotiate conditions for 
sharing data and could include payment to the 
user as a term, or even retract consent for the 
use of data previously authorized, for instance 

if a breach of conditions was detected. 


Nonetheless, the dominant paradigm for personal 
data models needs to shift away from system 
and service-based models not under the control 
of the individual/human, and toward a model 
focused on the individual. Personal data cannot 
be controlled or understood when fragmented 
and controlled by a myriad of entities in legal 
jurisdictions across the world. The object model 
for personal data should be associated with that 
person, and under the control of that person 
utilizing a personalized privacy A/IS or algorithmic 
guardian. 


During the handshake/negotiation between the 
personal agent and the system or service, the 
personal agent would decide what data to make 
available and under what terms, and the system 
would decide whether to make the service 
available, and at what level. If the required data 


set contains elements the personal agent will 
not provide, the service may be unavailable. 

If the recommended data set will not be 
provided, the service may be degraded. A user 
should be able to override his/her personal 
agents should he/she decide that the service 
offered is worth the conditions imposed. 


Vulnerable parts of the population will need 
protection in the process of granting access, 
especially given the asymmetry of power 
between an individual and entities. 


Candidate Recommendations 


Algorithmic guardian platforms should be 
developed for individuals to curate and share 
their personal data. Specifically: 


1. Such guardians could provide personal 
information control to users by helping 
them track what they have agreed to share 
and what that means to them, while also 
scanning each user's environment to set 
personal privacy settings accordingly. 


2. For purposes of privacy, a person must 
be able to set up complex permissions that 
reflect a variety of wishes. 


3. Default profiles, to protect naive or 
uninformed users, should provide little 
or no personal information without explicit 
action by the personal agent's owner. 


4. The agent should help a person foresee 
and mitigate potential ethical implications 
of specific machine learning data exchanges. 
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5. Control of the data from the agent should 
vest with the user, as otherwise users could 
lose access to his/her own ethical choices, 
and see those shared with third parties 
without permission. 


6. A guardian should enable machine-to- 
machine processing of information to 
compare, recommend, and assess offers 
and services. 


7. Institutional systems should ensure support 
and respect the ability for individuals to 
bring their own guardian to the relationship 
without any constraints that would make 
some guardians inherently incompatible 
or subject to censorship. 


Further Resources 


e The IEEE Global Initiative on Ethics of 
Autonomous and Intelligent Systems. 
Personal Data and Individual Access Control 
Section, in Ethically Aligned Design: A 
Vision for Prioritizing Human Well-being 
with Artificial Intelligence and Autonomous 
Systems, Version 1. IEEE, 2016. 


e IEEE P7006™, Standard for Personal Data 
Artificial Intelligence (Al) Agent was launched 
in the summer of 2017 and is currently in 
development. Readers of this section are 
encouraged to join the Working Group if they 
are focused on these issues. 








e We wish to acknowledge Jarno M. Koponen’s 
articles on Algorithmic Angels that provided 
inspiration for portions of these ideas. 
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e Companies are already providing solutions 
for early or partial versions of algorithmic 
guardians. Anonyome Labs recently 
announced their SudoApp that leverages 
strong anonymity and avatar identities 
to allow users to call, message, email, shop, 
and pay — safely, securely, and privately. 


e Tools allowing an individual to create 
a form of an algorithmic guardian are often 
labeled as PIMS, or personal information 
management services. Nesta in the United 
Kingdom was one of the funders of early 
research about PIMS conducted by CtrlShift. 











* Privacy Assistant from MIT. 








Issue: 


Consent is vital to information 
exchange and innovation in 

the algorithmic age. How can 

we redefine consent regarding 
personal data so it respects 
individual autonomy and dignity? 


Background 


Researchers have long identified some key 
problems with notice and consent in the digital 
world. First, individuals cannot and will not read 
all of the privacy policies and data use statements 
to which they are exposed, and even if they 
could, these policies are not easy to understand. 
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Individual consent is rarely exercised as a 
meaningful choice due to poorly provisioned 
user-appropriate design. 


A/IS place further strain on the notice and 
consent regime as further personalization of 
services and products should not be used as an 
excuse to minimize organizational transparency 
and choice for individuals to meet ethical and 
regulatory demand. If individuals opt not to 
provide personal information, they may find 
themselves losing access to services or receiving 
services based on stereotypes derived from 

the lower quality of data that they do provide. 


When consent is not feasible or appropriate, 
organizations should engage in a robust audit 
process to account for processing of personal 
data against the interests of individuals. For 
instance, the GDPR permits processing on the 
grounds of an entity's legitimate interests, so 
long as those interests do not outweigh the 


fundamental rights and interests of data subjects. 


Organizations must develop internal procedures 
for conducting such an analysis, and external 
actors and regulators should provide further 
guidance and oversight where possible. 


The needs of local communities, greater society, 
and public good should factor into this process. 
For example, a doctor may need medical data to 
be identified in order to treat a patient. However, 
a researcher may require it simply for statistical 
analysis, and therefore does not require the data 
to be identifiable. This is particularly important 


where the primary reason for data collection may 
mask important secondary uses post-collection. 
In time, however, new mechanisms for facilitating 
dynamic consent rules and core structure as use- 
cases change. As data moves from the original 
collection context to a change of context, agile 
ethics rules should be deployed. 


Candidate Recommendations 


The asymmetric power of institutions (including 
public interest) over individuals should not 

force use of personal data when alternatives 
such as personal guardians, personal agents, 
law-enforcement-restricted registries, and other 
designs that are not dependent on loss of agency 
are available. When loss of agency is required 

by technical expedience, transparency needs 

to be stressed in order to mitigate these 
asymmetric power relationships. 


Further Resources 


e Office of the Privacy Commissioner of 
Canada. “Consultation on Consent Under 
the ‘Personal Information Protection and 
Electronic Documents Act'.” September 
21, 2017. U.K. Information Commissioner's 
Office. “Consultation: GDPR Consent 


Guidance.” March 2017. 














e United Nations. “United Nations Declaration 
on the Rights of Indigenous Peoples.” 
107th plenary meeting, September 13, 2007. 
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Issue: 


Data that is shared easily 
or haphazardly via A/IS can 
be used to make inferences 
that an individual may not 
wish to share. 


Background 


It is common for a consumer to consent to the 
sharing of discrete, apparently meaningless data 
points like credit card transaction data, answers 
to test questions, or how many steps they walk. 
However, once aggregated these data and 

their associated insights may lead to complex 
and sensitive conclusions being drawn about 
individuals that consumers would not have 
consented to sharing. As analysis becomes more 
obfuscated via A/IS, not even data controllers 
will necessarily know what or how conclusions 
are being drawn through the processing of 
personal data, or how those data are used in 
the whole process. 


Opting out has some consequences. Users 

need to understand alternatives to consent 

to data collection before they give or withhold it, 
as meaningful consent. Without understanding 
the choices, consent cannot be valid. This places 
further strain on existing notions of informed 
consent. It raises the need for additional user 
controls and information access requirements. 

As computational power advances and algorithms 
compound existing data, information that was 


thought to be private or benign can be linked 

to individuals at a later time. Furthermore, this 
linked data may then be used to train algorithms, 
without transparency or consent, setting in 
motion unintended consequences. Auditing 

data use and collection for potential ethics risks 
will become increasingly more complex with 
A/IS in relation to these issues in the future. 


Candidate Recommendation 


The same A/IS that parses and analyzes data 
should also help individuals understand how 
personal information can be used. A/IS can 
prove granular-level consent in real time. Specific 
information must be provided at or near the 
point (or time) of initial data collection to provide 
individuals with the knowledge to gauge potential 
privacy risks in the long-term. Data controllers, 
platform operators, and system designers must 
monitor for consequences when the user has 
direct contact with an A/IS system. Positive, 
negative, and unpredictable impacts of accessing 
and collecting data should be made explicitly 
known to an individual to provide meaningful 
consent ahead of collection. Specifically: 


e — Terms should be presented in a way that 
allows the user to easily read, interpret, 
understand, and choose to engage with 
the system. To guard against these types 
of complexities, consent should be both 
conditional and dynamic. The downstream 
consequences (positive and negative) 
must be explicitly called out, such that the 
individual can make an informed choice, 
and/or assess the balance of value in context. 
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If a system impacts the ability of 

consumers to manage their own data via 
A/IS, accountability program management 
(PM) could be deployed to share consent 
solutions. A PM could span a diversity of 
tools and software applications to collect 

and transfer personal data. A PM can be 
assigned to evaluate consent metrics by 
ethics leadership to provide accountability 
reports. An actionable consent framework 

for personal data would not need to “reinvent 
the wheel.” Existing privacy and personal data 
metrics and frameworks can be integrated 
into consent program management, as it 
becomes relevant. Likewise, resources, user 
controls, and policies should be put in place 
to afford individuals the opportunity to retract 
or erase their data if they feel it is being used 
in ways they do not understand or desire. 
Use limitations are also important and may 
be more feasible than collection limitations. 
At a minimum, organizations should 

commit to not use data to make sensitive 
inferences or to make important eligibility 
determinations absent consent. Because 
consent is so challenging in A/IS, it is vital 
that user participation, including data access, 
erasure, and portability, are also incorporated 
into ethical designs. 


Moving all computational values to the 
periphery (on the person) seems to be the 
only way to combat all the risks articulated. 


Systems should be designed to enable 
personalization and meta system learning 
concurrently without the permanent 
collection and storage of personal data for 
retargeting. This is a key architectural design 
challenge that A/IS designers must achieve 
if Al is going to be of service to society. 


Further Resources 


Duhigg, C. “How Companies Learn Your 
Secrets.” The New York Times Magazine, 
February 19, 2012. 


Meyer, R. "When You Fall in Love, This Is 
What Facebook Sees.” The Atlantic, February 
15, 2014. 


Cormode, G. “The Confounding Problem 
of Private Data Release.” 18th International 
Conference on Database Theory (2015): 
1-12. 








Felbo, B., P. Sundsøy, A. Pentland, S. 
Lehmann, and Y. de Montjoye. “Using Deep 
Learning to Predict Demographics from 
Mobile Phone Metadata.” Cornell University 
Library, arXiv: 1511.06660, February 13, 2016. 


OECD Standard of Data Minimization — 
Minimum data required for maximum 
service. 
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Issue: 


Many A/IS will collect data 

from individuals they do not 
have a direct relationship with, 

or the systems are not interacting 
directly with the individuals. 

How can meaningful consent 

be provided in these situations? 


Background 


Individuals can be better informed of uses, 
processing, and risks of data collection when 
they interact with a system. loT presents evolving 
challenges to notice and consent. Data subjects 
may not have an appropriate interface to 
investigate data controller uses and processes. 
They may not be able to object to collection 

of identifiable information, known or unknown 

to them by wireless devices, driven by A/IS. 


When individuals do not have a relationship 

with the data collecting system, they will have 
no way of participating in their data under the 
notice and consent regime. This challenge is 
frequently referenced as the “Internet of Other 
People's Things.” A/IS embodied in loT devices 
and value-chains will need better interfaces and 
functionality to help subjects understand and 
participate in the collection and use of their data. 
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Candidate Recommendations 


Where the subject does not have a direct 
relationship with the system, consent should 

be dynamic and must not rely entirely on initial 
terms of service or other instruction provided 
by the data collector to someone other than the 
subject. A/IS should be designed to interpret 
the data preferences, verbal or otherwise, of all 
users signaling limitations on collection and use, 
discussed further below. 


Further Resources 


e Kaminski, M. “Robots in the Home: What 
Will We Have Agreed To?” Idaho Law Review 
51, no. 661 (2015): 551-677. 


e Jones, M. L. “Privacy Without Screens and 
the Internet of Other People’s Things,” Idaho 
Law Review 51, no. 639 (2015): 639-660. 


e Cranor, L. F. “Personal Privacy Assistants in 
the Age of the Internet of Things,” presented 
at the World Economic Forum Annual 
Meeting, 2016. 
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Issue: 


How do we make better 
user experience and consent 
education available to 
consumers as standard to 
express meaningful consent? 


Background 


Individuals are often not given agency or 
personal tools to express, invoke, or revoke 
consent to the terms of service or privacy and/ 
or data use policies in their contracts. In many 
cases, individual data subjects were not notified 
at all of the transfer of their data in the course 
of business or government exchanges. 


Industry data uses have led to individual 
exposure to intangible and tangible privacy 
harms, for example, mistaken identity. Inability 
to manage or control information has also 

led to barriers to employment, healthcare, and 
housing. This dynamic has resulted in some 
consumer resignation over the loss of control 
over personal information, despite a stated 
desire for additional control. 


Candidate Recommendations 


Tools, settings, or consumer education are 
increasingly available and should be utilized to 
develop, apply, and enforce consumer consent. 
Specifically: 


Design the terms of service (ToS) as 
negotiable to consumers — Combine 

user interface design to control the rate 

and method of data exchange, and provide 

a corporate terms ombudsman staffed as 
human agency to consumers facing a terms 
of service contract. Software developers 
would produce contract management 
platforms appropriate for consumer 
negotiation. This would support features to 
negotiate terms of consent contracts fairly for 
meaningful consumer consent. An example 
metric would be a consumer agreement 
held to 85% of a terms of service agreement 
content, as grounds to move forward with 
the contract. Companies conclude what the 
"deal breakers” or non-negotiables are ahead 
of time. 


Provide “privacy offsets” as a business 
alternative to the personal data 
exchange — Provide a pay alternative to the 
freemium data exchange model, to limit 

or cap third party vendor access to personal 
data or limit transactional data to internal 
business use only. Business developers 
would have to cost count individual data 
based on a general market profile, or offer 

a flat rate for advertising-free service. If they 
know immediately how much money they 
will lose if a new user would not consent to 
an external data exchange, they have grounds 
to pass the cost to new consumers as a 
privacy offset product. 
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¢ Apply “consent” to further certify 
artificial intelligence legal and as 
ethics doctrine — Legal consent principles 
could be applied to a larger self-regulatory 
or co-regulatory artificial intelligence ethics 
certification framework for businesses 
and governments. This would be similar 
to medical certifications in ethics as a 
professional requirement, supportive of 
the Hippocratic Oath. Artificial intelligence 
ethics certification for responsible 
institutions (medical, government, education, 
corporations) should include education in 
applied legal consent principles, situation 
training regarding forms of consent, ethics 
certification testing, and perhaps a notarized 
public declaration to uphold ethical principles 
of consent. As an ethics board is formed it 
might: evaluate complaints, resolve ethical 
conflicts related to artificial intelligence and 
consent issues, improve upon current ethics 
procedures for consent, request independent 
investigations, review licensure or certification 
determinations, recommend professional 
penalties or discipline to organizations, 
and/or file legal claims based on findings. 
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Aggregate and provide visualization 
options for terms of service and privacy 
statements — One way to provide better 
education and improved user experience, 
with respect to legal terms of use, is to offer 
visual analytics tools as a consumer control 
point of reference. Potential examples 

of this sort of effort include the Terms of 
Service Didn't Read Project and the Clarip. 
Both tools simplify the content of these 
policies and may provide users with clarity 
into how services are collecting, making 

use of, and potentially sharing personal and 
other information. 





Further Resources 


Cavoukian, A. "Privacy by Design: The 7 
Foundational Principles. Implementation 
and Mapping of Fair Information Practices.” 
Internet Architecture Board, 2010. 











"From Consent to Data Control by Design.” 
Data Ethics, March 20, 2017. 





Hintze, M. Privacy Statements: Purposes, 
Requirements, and Best Practices. 








Cambridge, U.K.: Cambridge University 
Press, 2017. 
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Issue: 


In most corporate settings, 
employees do not have clear 
consent on how their personal 
information (including health 
and other data) is used by 
employers. Given the power 
differential between employees 
and employers, this is an area in 
need of clear best practices. 


Background 


In the beginning stages of onboarding, many 
employees sign hiring agreements that license 
or assign the usage of their data in very non- 
specific ways. This practice needs to be updated, 
so that it is clear to the employee what data is 
collected, and for what purpose. The employee 
must also have the ability/possibility to request 
privacy for certain data as well as have the 

right to remove the data if/when leaving the 
employment. 


Candidate Recommendation 

In the same way that companies are doing 
privacy impact assessments for how individual 
data is used, companies need to create employee 
data impact assessments to deal with the 
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specific nuances of corporate specific situations. 
It should be clear that no data is collected 
without the consent of the employee. 


Furthermore, it is critical that the data: 


Is gathered only for specific, explicitly 
stated, and legitimate purposes 


Is correct and up to date 
Is only processed if it is lawful 


Is processed in a proper manner, 
and in accordance with good practice 


Is not processed for any purpose that 
is incompatible with that for which the data 
was gathered 


Is rectified, blocked, or erased if it is 
incorrect or incomplete having regard 
for the purpose of the processing 


Is not kept for a longer period than 
is necessary 


Further Resources 


The Swedish Personal Data Protection Act 

is taking a generic approach to data protection 
and data privacy, but it is well applicable 

for the specific case of employee data. 





IEEE P7005™, Standard for Transparent 
Employer Data Governance. This Working 
Group is open and free for anyone to join. 
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People may be losing their 
ability to understand what kinds 
of processing is done by A/IS 
on their private data, and thus 
may be becoming unable to 
meaningfully consent to online 
terms. The elderly and mentally 
impaired adults are vulnerable 
in terms of consent, presenting 
consequence to data privacy. 


Background 


The poor computer literacy of the elderly has 
been well known from the beginning of the 
information and Internet age. Among various 
problems related to this situation, is the financial 
damage caused by the misuse of their private 
information, possibly by malicious third parties. 
This situation is extremely severe for elderly 
people suffering from dementia. 
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Candidate Recommendations 


Researchers or developers of A/IS have 

to take into account the issue of vulnerable 
people, and try to work out an A/IS that 
alleviates their helpless situation to prevent 
possible damage caused by misuse of their 
personal data. 


Build an Al advisory commission, composed 
of elder advocacy and mental health self- 
advocacy groups, to help developers produce 
a level of tools and comprehension metrics 
to manifest meaningful and pragmatic 
consent applications. 
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Autonomous systems designed to cause physical harm have additional ethical dimensions 
as compared to both traditional weapons and autonomous systems not designed to cause 
harm. Multi-year discussions on international legal agreements around autonomous systems 
in the context of armed conflict are occurring at the United Nations (UN), but professional 
ethics about such systems can and should have ethical standards covering a broad array 

of issues arising from the automated targeting and firing of weapons. 


Broadly, we recommend that technical organizations promote a number of measures 
to help ensure that there is meaningful human control of weapons systems: 


e That automated weapons have audit trails to help guarantee accountability 
and control. 


e That adaptive and learning systems can explain their reasoning and decisions 
to human operators in transparent and understandable ways. 


e That there be responsible human operators of autonomous systems who are 
clearly identifiable. 


e That the behavior of autonomous functions should be predictable to their operators. 
e That those creating these technologies understand the implications of their work. 


e That professional ethical codes are developed to appropriately address the 
development of autonomous systems and autonomous systems intended 
to cause harm. 


Specifically, we would like to ensure that stakeholders are working with sensible and 
comprehensive shared definitions, particularly for key concepts relevant to autonomous 
weapons systems (AWS). Designers should always ensure their designs meet the standards 
of international humanitarian law, international human rights law, and any treaties or 
domestic law of their particular countries, as well as any applicable engineering standards, 
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military requirements, and governmental regulations. We recommend designers not only 
take stands to ensure meaningful human control, but be proactive about providing quality 
situational awareness to operators and commanders using those systems. Professional 
ethical codes should be informed by not only the law, but an understanding of both local- 
and global-level ramifications of the products and solutions developed. This should include 
thinking through the intended use or likely abuse that can be expected by users of AWS. 


While the primary focus of this document is with kinetic AWS that cause physical harm, it 

is recognized that many of these concerns and principles may also apply to cyber-weapons. 
This is, of course, also pertinent to cyber-weapons that have kinetic effects, such as those 
that destroy civilian infrastructures or turn civilian objects, vehicles, or infrastructure into 
kinetic weapons. 


Additionally, society must be aware of the variety of political and security threats posed 
by AWS. Miniaturized AWS will pose additional threats because they are small, insidious, 
or obfuscated, and may therefore be non-attributable to the deploying entity. Depending 
upon payload or weapons (such as chemical, biological, or nuclear weapons), these may 
autonomously deploy weapons of mass destruction (WMD), or themselves constitute 

a new form of WMD. Additional ethical recommendations are needed to prevent the 
development of systems having these dangerous properties. 


e Issues 1-3 raise general high-level questions regarding the definition of AWS 
and their relation to existing law and ethics. 


e Issues 4—10 raise socio-political concerns over the likely uses and effects 
of AWS development and use. 


e Issue 11 raises engineering concerns over the specific challenges posed 
by autonomous systems capable of targeting and deploying weapons. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Issue 1: 


Confusions about definitions 
regarding important concepts 

in artificial intelligence (Al), 
autonomous systems (AS), and 
autonomous weapons systems 
(AWS) stymie more substantive 
discussions about crucial issues. 


Background 


The potential for confusion about AWS definitions 
is not just an academic concern. The lack of clear 
definitions regarding what constitutes AWS is often 
cited as a reason for not proceeding toward any 
kind of international governance over autonomous 
weapons. As this is both a humanitarian issue 

and an issue of geopolitical stability, the focus 

in this area needs to be on how the weapons 

are controlled by humans rather than about the 
weapons’ technology per se. 


The term autonomy is important for 
understanding debates about AWS; yet there 
may be disputes — about what the term means 
and whether what the definition identifies is 
technically possible today. This prevents progress 
in developing appropriate policies to regulate 
AWS design, manufacture, and deployment. 
Consistent and standardized definitions are 
needed to enable effective discussions of AWS, 
but they should be general enough to enable 
flexibility to ensure that those definitions do not 
become quickly technologically outdated. 
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Moreover, the phrases “human in the loop” and 
“human on the loop" also lack clarity and only 
contribute further confusion. Depending upon 
what one means, “in the loop” or “on the loop” 
means different things to different people. It 
could be used to describe the command chain 
that authorizes weapon release, where the 
commands flow down to a human and a weapon 
system to take specific actions. Yet, there are 
micro-level decisions where a human operator 
may have an opportunity to question the 
command. What often matters is the time delay 
between the fielding of an autonomous system, 
the decision to engage a weapon against a target, 
and the impact time. 


Contrarily, “in the loop” obscures another 
temporal question: that whether in these 
scenarios clearance to fire at a target entails an 
authorization to prosecute that target indefinitely, 
or whether there are necessarily predetermined 
limits on the amount of time or ordinance 

each clearance provides. Central to this issue 

is how long a target that has been designated 
and verified by an authorized human in a given 
situational context remains a legitimate target. 


This notion of autonomy can be applied 
separately to each of the many functions of a 
weapons system; thus, an automatic weapons 
system could be autonomous in searching 

for targets, but not in choosing which ones 

to attack, or vice versa. It may or may not be 
given autonomy to fire in self-defense when 
the program determines that the platform is 
under attack, and so on. Within each of these 
categories, there are also many intermediate 
gradations in the way that human and machine 
decision-making may be coupled. 
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Candidate Recommendations 


The term autonomy in the context of AWS should 
be understood and used in the restricted sense 
of the delegation of decision-making capabilities 
to a machine. Since different functions within 
AWS may be delegated to varying extents, and 
the consequences of such delegation depend 

on the ability of human operators to forestall 
negative consequences via the decisions over 
which they retain effective control, it is important 
to be precise about the control of specific 
functions delegated to a given system, as well as 
the ways in which control over those functions 
are shared between human operators and AWS. 


We support the working definition of AWS 
offered by the International Committee of the 
Red Cross (ICRC) and propose that it be adopted 
as the working definition of AWS for the further 
development and discussion of ethical standards 
and guidelines for engineers. The ICRC defines 
an AWS as: “any weapon system with autonomy 
in its critical functions. That is, a weapon system 
that can select (i.e. search for or detect, identify, 
track, select) and attack (i.e. use force against, 
neutralize, damage or destroy) targets without 
human intervention.” 


Further Resources 


e Dworkin, G. The Theory and Practice of 
Autonomy. Cambridge, U.K.: Cambridge 
University Press, 1988. 


e Frankfurt, H. G. “Freedom of the Will and the 
Concept of a Person,” in The Importance 
of What We Care About, Cambridge, U.K.: 
Cambridge University Press, 1987. 


¢ DoD Defense Science Board, The Role 
of Autonomy in DoD Systems, Task Force 
Report. July 2012, 48. 
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DoD Defense Science Board, Summer Study 
on Autonomy. June 2016. 


Young, R. Autonomy: Beyond Negative 
and Positive Liberty. New York: St. Martin's 
Press, 1986. 


Society of Automotive Engineers. J3016, 
Taxonomy and Definitions for Terms Related 
to On-Road Motor Vehicle Automated Driving 
Systems. SAE International, 2014. 


Roff, H. M. “An Ontology of Autonomy: 
Autonomy in Weapons Systems,” in The 
Ethics of Autonomous Weapons, edited 
by C. Finkelstein, D. MacIntosh, and J. D. 
Ohlin. Cambridge, U.K.: Oxford University 
Press, forthcoming. 


Sharkey, N. “Towards a Principle for the 
Human Supervisory Control of Robot 
Weapons.” Politica and Societa 2 (2014): 
305-324. 


U.K. Ministry of Defence. UK Joint Doctrine 
Note (JDN) 3/10, “Unmanned Aircraft 
Systems: Terminology, Definitions and 
Classification.” May 2010. 


U.K. Ministry of Defence. UK Joint Doctrine 
Note (JDN) 2/11, “The UK Approach to 
Unmanned Aircraft Systems.” March 2011. 


United Nations Institute for Disarmament 
Research (UNIDIR). “Framing Discussions 
on the Weaponization of Increasingly 
Autonomous Technologies.” 2014 


International Committee of the Red Cross 
(ICRC). “Autonomous Weapon Systems: 
Implications of Increasing Autonomy in 
the Critical Functions of Weapons.” 
September 1, 2016. 
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Issue 2: 


The addition of automated 
targeting and firing functions to 
an existing weapon system, or 
the integration of components 
with such functionality, or system 
upgrades that impact targeting 
and automated weapon release 
should be considered for review 
under Article 36 of Additional 
Protocol | of the Geneva 
Conventions. 


Background 


According to Article 36 of Additional Protocol 

| to the Geneva Conventions (1977), “In the 
study, development, acquisition or adoption of 
a new weapon, means or methods of warfare,” 
weapon systems must be internally reviewed 
for compliance with international humanitarian 
law (IHL). Alterations to the critical functions or 
targeting and weapons release of an already- 
reviewed weapons systems should be considered 
for review, and any system automating those 
functions should be reviewed to ensure 
meaningful human control. 








International human rights law (IHRL) also 
guarantees, by way of international and bilateral 
treaties, rights to life, human dignity, fair trial, 
and further positive and negative human rights. 
Society and engineers must consider the ways 
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in which these rights may be threatened by the 
deployment and/or use of AWS, during armed 
conflict, policing, or other security operations. 


There are situational and operational limitations 
of all engineered systems, and complete 
knowledge is not something that can be expected 
or required. However, there must be a multi-level 
effort to: 


e Evaluate the conformity of a system to the law 


e Evaluate its reliability and applicability for 
a given mission 


e Evaluate its ability to conform to rules 
of engagement 


Further, key decision makers need to 
understand the engineering constraints and 
limitations of weapons systems with high 
degrees of autonomy. 


Candidate Recommendations 


e All engineering work should conform to the 
requirements of international law, including 
both IHL and IHRL, as well as national and 
local laws. While this is not the primary 
responsibility of an individual engineer, 
there ought to be opportunities for engineers 
to learn about their obligations, their 
responsibilities with respect to AWS, 
as well as keeping their employing 
agencies accountable. 


e Meaningful human control over the critical 
functions in weapons systems can help 
ensure that weapons can be used in 
conformity with the law in each instance. It is 
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also necessary for all stakeholders to consider 
design and implement accountability 
measures to help ensure all weapons are 
used in conformity with the law. 


+ Engineering constraints should be clearly 
identified, defined, and communicated to 
Article 36 weapons reviewers, to operators 
in their training for a system, and to military 
commanders and their legal counsel charged 
with specifying the rules of engagement. 


e All those with responsibilities for weapon 
systems should ensure that Article 36 
reviews will be held and provide all evidence 
needed at them. This should include any 
data which will lead to restrictions on their 
use, which will also be needed for Article 36 
reviews and for military staff to set rules of 
engagement for the weapon system's use. 


+ There should be greater engineering input 
into the weapons reviews, and greater 
communication between engineers and 
lawyers in the weapons review process 
to ensure meaningful human control 
over weapons. 


Further Resources 


e — International Committee of the Red Cross 
(ICRC). “Autonomous Weapon Systems: 
Implications of Increasing Autonomy 
in the Critical Functions of Weapons.” 
September 1, 2016. 
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Issue 3: 


Engineering work should conform 
to individual and professional 
organization codes of ethics and 
conduct. However, existing codes 
of ethics may fail to properly 
address ethical responsibility for 
autonomous systems, or clarify 
ethical obligations of engineers 
with respect to AWS. Professional 
organizations should undertake 
reviews and possible revisions 

or extensions of their codes of 
ethics with respect to AWS. 


Background 


e The ethical requirements for engineering 
have an independent basis from the 
law, although they are hopefully aligned 
with written laws and written codes of 
professional ethics. Where agreed upon, 
ethical principles are not reflected in 
written laws and ethical codes, individuals 
and organizations should strive to correct 
those gaps. 


e — Ethical requirements upon engineers 
designing autonomous weapon systems 
may go beyond the requirements of meeting 
local, national, and international laws. 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 118 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Reframing Autonomous Weapons Systems 


© 


creative 
commons 


Many professional organizations have codes of 
conduct intended to align individuals’ behaviors 
toward particular values. However, they seldom 
sufficiently address members’ behaviors in 
contributing toward particular artifacts, such 

as creating technological innovations deemed 
threatening to humanity, especially when those 
innovations have significant probabilities of costly 
outcomes to people and society. Foremost 
among these in our view are technologies related 
to the design, development, and engineering 

of AWS. 


Organizations such as the IEEE, the Association 
for Computing Machinery (ACM), the Association 
for the Advancement of Artificial Intelligence 
(AAAI), the UK Royal Academy of Engineering, 
the Engineering Council, Engineers Canada, and 
the Japanese Society for Artificial Intelligence 
(SAI) have developed codes of ethics. Some of 
these groups are currently reviewing those codes 
in light of current and future developments 

in autonomous systems and Al. 


While national laws may differ on what 
constitutes responsibility or liability for the design 
of a weapon system, given the level of complicity 
or the causal contribution to the development 

of a technology, ethics looks for lines of moral 
responsibility. Determining whether an individual 
is morally responsible requires understanding the 
organizations in which they work and to establish 
relevant facts in relation to the individual’s acts 
and intentions. 


Candidate Recommendations 


Codes of conduct should be extended to govern 
a member's choice to create or contribute to 
the creation of technological innovations that 
are deemed threatening to humanity. Such 
technologies carry with them a significant 
probability of costly outcomes to people and 
society. When codes of conduct are directed 
toward ensuring positive benefits or outcomes 
for humanity, organizations should ensure 

that members do not create technologies that 
undermine or negate such benefits. In cases 
where created technologies or artifacts fail to 
embody or conflict with the values espoused in a 
code of conduct, it is imperative that professional 
organizations extend their codes of conduct 

to govern these instances so members have 
established recourse to address their individual 
concerns. Codes of conduct should also more 
broadly ensure that the artifacts and agents 
offered into the world by members actively 
reflect the professional organization's standards 
of professional ethics. 


Professional organizations need to have resources 
for their members to make inquiries concerning 
whether a member's work may contravene (IHL) 
or (IHRL). 


How one determines the line between ethical 
and unethical work on AWS requires that one 
address whether the development, design, 
production, and use of the system under 
consideration is itself ethical. It is incumbent 
upon a member to engage in reflective 
judgment to consider whether or not his or 
her contribution will enable or give rise to AWS 
and their use cases. Members must be aware 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 








119 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Reframing Autonomous Weapons Systems 





of the rapid, dynamic, and often escalatory 
natures of interactions between near-peer 
geopolitical adversaries or rivals. It is also 
incumbent upon members of a relevant technical 
organization to take all reasonable measures 

to inform themselves of the funding streams, 

the intended use or purpose of a technology, 
and the foreseeable misuse of their technology 
when their contribution is toward AWS in whole 
or in part. If their contribution to a system is 
foreseeably and knowingly to aid in human-aided 
decisions — that is, as part of a weapon system 
that is under meaningful human control — this 
may act as a justification for their research. 


Further Resources 


e Kvalnes, Ø. “Loophole Ethics,” in Moral 
Reasoning at Work: Rethinking Ethics in 
Organizations, 55—61. Palgrave Macmillan 
U.K., 2015. 


e Noorman, M. “Computing and Moral 
Responsibility,” The Stanford Encyclopedia 
of Philosophy, edited by Edward N. Zalta , 
Summer 2014 Edition. 


e Hennessey, M. “Clearpath Robotics Takes 


Stance Against ‘Killer Robots’.” Clearpath 
Robotics, 2014. 


« “Autonomous Weapons: An Open Letter from 
Al & Robotics Researchers.” Future of Life 
Institute, 2015. 


e Noorman, M. “Computing and Moral 
Responsibility,” in The Stanford Encyclopedia 
of Philosophy (Summer 2014 Edition), edited 
by Edward N. Zalta. 


* “Engineers Canada Code of Ethics,” 2017. 
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e The Japanese Society for Artificial Intelligence 
Ethical Guidelines, 2017 








* Engineering Council and Royal Academy of 
Engineering, Statement of Ethical Principles 
for the Engineering Profession. 











Issue 4: 


The development of AWS 

by states is likely to cause 
geopolitical instability and could 
lead to arms races. 


Background 


The widespread adoption of AWS by nation states 
could present a unique risk to the stability of 
international security. Because of the advantages 
of either countering an adversary through 
concomitant adoption of arms or being the 

first or prime mover is an offset advantage, the 
pursuit of AWS is likely to spur an international 
arms race. Evidence of states seeking greater 
adoption of artificial intelligence and quantum 
computing for security purposes already 

exists. The deployment of machine learning 

and other artificial intelligence applications on 
weapons systems is not only occurring, but 

will continue to advance. Thus it is important 

to look to previous scholarship on arms race 
dynamics to be informed about the first- and 
second-order effects of these races, such as the 
escalatory effects, arms development, decreasing 
international stability, and arms proliferation. 
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Candidate Recommendations 


Autonomous weapons designers should support 
the considerations of the United Nations to 
adopt a protocol to ensure meaningful human 
control over AWS under the Convention on 
Certain Conventional Weapons (CCW) treaty, or 
other similar effort by other international bodies 
seeking a binding international treaty. 


It is unethical to design, develop, or engineer 
AWS without ensuring that they remain reliably 
subject to meaningful human control. Systems 
created to act outside of the boundaries of 
“appropriate human judgment,” “effective human 
control,” or “meaningful human control,” violate 
fundamental human rights and undermine legal 
accountability for weapons use. Various scenarios 
for maintaining meaningful human control over 
weapons with autonomous functions should 

be further investigated for best practices by a 
joint workshop of stakeholders and concerned 
parties (including, but not limited to, engineers, 
international humanitarian organizations, and 
militaries), and that those best practices be 
promoted by professional organizations as 

well as by international law. 


Further Resources 


* Scharre, P, and K. Sayler. “Autonomous 
Weapons and Human Control” (poster). 
Center for a New American Security, 
April 2016. 


e International Committee for Robot Arms 
Control. “LAWS: Ten Problems for Global 
Security” (leaflet). April 10, 2015. 


* — Roff, H. M., and R. Moyes. “Meaningful 
Human Control, Artifical Intelligence and 
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Autonomous Weapons.” Briefing paper 
prepared for the Informal Meeting of Experts 
on Lethal Autonomous Weapons Systems, 
UN Convention on Certain Conventional 
Weapons, April 2016. 





e — United Nations Institute for Disarmament 
Research (UNIDIR). “The Weaponization 
of Increasingly Autonomous Technologies: 
Considering How Meaningful Human Control 
Might Move the Discussion Forward." 2014. 

















Issue 5: 


The automated reactions of an 
AWS could result in the initiation 
or escalation of conflicts outside 
of decisions by political and 
military leadership. AWS that 
engage with other AWS could 
escalate a conflict rapidly, before 
humans are able to intervene. 


Background 


One of the main advantages cited regarding 
autonomous weapons is that they can make 
decisions faster than humans, enabling rapid 
defensive and offensive actions. When opposing 
autonomous weapons interact with each other, 
conflict might escalate without explicit human 
military or political decisions, and escalate more 
quickly than humans on either side will be able to 
understand or act. 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 121 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Reframing Autonomous Weapons Systems 


© 


creative 
commons 


Candidate Recommendations 


e — Consider ways of limiting potential harm from 
automated weapons. For example: limited 
magazines, munitions, or maximum numbers 
of platforms in collaborative teams. 


+ — Explore other technological means for limiting 
escalation, for example, “circuit breakers,” as 
well as features that can support confidence- 
building measures between adversaries. 

All such solution options ought to precede 
the design, development, deployment, and 
use of weapons systems with automated 
targeting and firing functions. 


e Perform further research on how to temper 
such dynamics when designing these 
systems. 


Further Resources 


* Scharre, P. “Autonomous Weapons and 
Operational Risk.” Washington, DC: Center 
for New American Security, February, 
2016. 











Issue 6: 


There are multiple ways in which 
accountability for the actions of 
AWS can be compromised. 


Background 


Weapons may not have transparency, auditability, 
verification, or validation in their design or use. 
Various loci of accountability include those for 
commanders (e.g., what are the reasonable 
standards for commanders to maintain 
meaningful human control?), and operators (e.g., 
what are the levels of understanding required by 
operators to have knowledge of the system state, 
operational context, and situational awareness?). 


Ideally all procurers, suppliers, and users 

of weapons systems components have 
accountability for their part of every weapons 
system, potential incorporation in future systems, 
and expected and potential users. 


Candidate Recommendations 


* Designers should follow best practices in 
terms of design process, which entails clearly 
defined responsibilities for organizations, 
companies, and individuals within the 
process. 


* — Systems and components should be 
designed to deter the easy modification of 
the overall weapon after the fact to operate 
in fully autonomous mode. 


e — Further exploration of black box recording 
of data logs, as well as cryptographic, block- 
chain, and other technical methods for 
tracing access and authorization of weapons 
targeting and release is needed. 
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* System engineers must work to the same 
high standards and regulations of security for 
AWS design from a cybersecurity perspective 
than they would for any other work. 
Weapons systems ought to be designed with 
cybersecurity in mind such that preventing 
tampering, or at least undetected tampering, 
is a highly weighted design constraint. 


* Procurement authority: only contract with 
contractors who have proper legal and 
security processes; carry out Article 36 
reviews at all major steps in the procurement 
process; maintain database of design, tests, 
and review evidence. 


e Contractors: ensure design meets relevant 
engineering and defense standards for 
military products; deliver evidence for Article 
36 reviews using, but not restricted to, design 
reviews and simulation models; provide 
evidence requested by user for setting ROE; 
ensure design has clear criteria for decisions 
made by their product. 


e Acceptance body: have validation and test 
plans for behavior of actual system produced; 
test weapons systems in a number of 
representative scenarios; have plans to 
ensure upgrades are reviewed against IHL 
criteria such as Article 36. 


* User/military commanders: only operate 
weapons systems with meaningful human 
control and in accordance with delegated 
authority. 
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* Weapons systems must have default modes 
of operation agreed with campaign planners 
before operation commences. 


e Ensure as many aspects of weapons systems 
as possible are designed with fail-safe 
behaviors. 


* Ensure clear embedded lines of 
accountability in the design, deployment, and 
operation of weapons. 


e Trusted user authentication logs and audit 
trail logs are necessary, in conjunction 
with meaningful human control. Thorough 
human-factors-driven design of user 
interface and human—computer/robot 
interaction design is necessary for situational 
awareness, knowability, understandability, 
and interrogation of system goals, reasons, 
and constraints, such that the user could be 
held culpable. 


* Tamper-proof the equipment used to store 
authorization signals and base this on open, 
auditable designs, as suggested by Gubrud 
and Altmann (2013). Further, the hardware 
that implements the human-in-the-loop 
requirement should not be physically distinct 
from operational hardware. 


There will need to be checks that all these 
bodies and organizations have discharged 

their responsibilities according to IHL and their 
domestic laws. Even if this is the case, weapons 
system operations may be compromised by, 

for example, equipment failure, actions by 
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opponents such as cyber-attacks, or deception 
so that the automated functions act according to 
design but against an incorrect target. 


There are currently weapons systems in use that, 
once activated, automatically intercept high-speed 
inanimate objects such as incoming missiles, 
artillery shells, and mortar grenades. Examples 
include SEA-RAM, C-RAM, Phalanx, NBS Mantis, 
and Iron Dome. These systems complete their 
detection, evaluation, and response process 
within a matter of seconds and thus render 

it extremely difficult for human operators to 
exercise meaningful supervisory control once 
they have been activated, other than deciding 
when to switch them off. This is called supervised 
autonomy by the U.S. Department of Defense 
(DoD) because the weapons require constant 
and vigilant human evaluation and monitoring 

for rapid shutdown in cases of targeting errors, 
change of situation, or change in status of targets. 
However, most of these systems are only utilized 
in a defensive posture for close-in weapons 
systems support against incoming lethal threats. 


Further Resources 


e  Gubrud, M., and J. Altmann. “Compliance 
Measures for an Autonomous Weapons 
Convention.” International Committee for 
Robot Arms Control, 2013. 





e U.K. Ministry of Defence. “The UK Approach 
to Unmanned Aircraft Systems (UAS),” Joint 
Doctrine Note 2/11, March 2011. 


« Sharkey, N. “Towards a Principle for the 
Human Supervisory Control of Robot 
Weapons.” Politica and Societa 2 (2014): 
305-324. 


Owens, D. “Figuring Forseeability.” Wake 
Forest Law Review 44 (2009): 1277, 
1281-1290. 


Roff, H. M., and R. Moyes. “Meaningful 
Human Control, Artificial Intelligence and 
Autonomous Weapons Systems.” Briefing 
Paper for the Delegates at the Convention 
on Certain Conventional Weapons Meeting 
of Experts on Lethal Autonomous Weapons 
Systems, Geneva, April 2016. 








Roff, H. M. “Meaningful Human Control or 
Appropriate Human Judgment.” Briefing 
Paper for the Delegates at the 5th Review 
Conference at the Convention on Certain 
Conventional Weapons, Geneva, December 
2016. 








Scherer, M. “Who's to Blame (Part 4): Who's 





to Blame if an Autonomous Weapon Breaks 
the Law?” Law and Al, February 24, 2016. 





Rebecca C, “War Torts: Accountability 
for Autonomous Weapons.” University of 








Pennsylvania Law Review 164, no. 6 (2016): 


1347-1402. 


Gillespie, T., and R. West. “Requirements for 
Autonomous Unmanned Air Systems Set by 
Legal Issues.” International C2 Journal 4, no. 
2 (2010): 1-32. 


Defense Science Board. “Summer Study on 
Autonomy.” Washington, DC: Office of the 
Under Secretary of Defense for Acquisition, 
Technology and Logistics, June 2016. 





Rickli, J.-M. “Artificial Intelligence and the 
Future of Warfare” (Box 3.2.1). 2017 Global 
Risk Report, Geneva: World Economic 
Forum, 2017. 
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Issue 7: 


AWS offer the potential for severe 
human rights abuses. Exclusion 
of human oversight from the 
battlespace can too easily lead 

to inadvertent violation of human 
rights. AWS could be used 

for deliberate violations of 
human rights. 


Background 


The ethical disintermediation afforded by AWS 
encourages the bypassing of ethical constraints 
on people's actions that should require the 
consent of multiple people, organizations, or 
chains of commands. This exclusion concentrates 
ethical decision-making into fewer hands. 


The potential lack of clear lines of accountability 
for the consequences of AWS might encourage 
malicious use of AWS by those seeking to avoid 
responsibility for malicious or illegal acts. 


Candidate Recommendations 


Acknowledge that the design, development, or 
engineering of AWS for anti-personnel or anti- 
civilian purposes are unethical. An organization's 
values on respect and the avoidance of harm 

to persons precludes the creation of AWS that 
target human beings. If a system is designed 
for use against humans, such systems must be 
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designed to be semi-autonomous, where the 
control over the critical functions remains with a 
human operator, (such as through a human-in- 
the-loop hardware interlock). Design for operator 
intervention must be sensitive to human factors 
and intended to increase, rather than decrease, 
situational awareness. 


Under no circumstances is it morally permissible 
to use AWS without meaningful human control, 
and this should be prohibited. Ultimately, 
weapons systems must be under meaningful 
human control. As such, design decisions 
regarding human control must be made so 

that a commander has meaningful human 
control over direct attacks during the conduct of 
hostilities. In short, this requires that a human 
commander be present and situationally aware of 
the circumstances on the ground as they unfold 
to deploy either semi-autonomous or defensive 
anti-materiel AWS. Organizational members must 
ensure that the technologies they create enhance 
meaningful human control over increasingly 
sophisticated systems and do not undermine 

or eliminate the values of respect, humanity, 
fairness, and dignity. 


Further Resources 


* Heller, K. J. “Why Preventive Self-Defense 
Violates the UN Charter.” Opinio Juris, March 
7, 2012. 








* Scherer, M. “Who's to Blame (Part 5): A 
Deeper Look at Predicting the Actions of 
Autonomous Weapons.” Law and Al, February 
29, 2016. 
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e Roff, H. M. “Killer Robots on the Battlefield: 
The Danger of Using a War of Attrition 
Strategy with Autonomous Weapons.” S/ate, 
2016. 











e Roff, H. “Autonomous Weapons and 
Incentives for Oppression.” Duck of Minerva, 
March 13, 2016. 








AWS could be used for 
covert, obfuscated, and 
non-attributable attacks. 


Background 


The lack of a clear owner of a given AWS 
incentivizes scalable covert or non-attributable 
uses of force by state and non-state actors. 
Such dynamics can easily lead to unaccountable 
violence and societal havoc. 


Features of AWS that may contribute to their 
making covert and non-attributable attacks easier 
include: small size; the ability to swarm; and 
ability to act at great distance and time from 

the deployment of a weapon from responsible 
operators; layers of weapons systems within 
other systems. 


States have a legal obligations to make attacks 
practically attributable. There are additional 
legal obligations not to booby trap autonomous 
systems. Self-destructive functions, such as 
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those aimed at preventing access to sensitive 
technologies or data, should be designed to not 
cause incidental or intentional harm. 


There are significant concerns about the use of 
AWS by non-state actors, or individuals, and the 
potential for use in terror attacks against civilians, 
and non-attributable attacks against states. 
Designers should be concerned about 

the potential of systems to be used by 

malicious actors. 


Candidate Recommendation 


Because AWS are delegated authority to use 
force in a particular situation, they are required 
to be attributable to the entity and human that 
deployed them. Designers should ensure that 
there is a clear and auditable authorization of 
actions taken by the AWS when in operation. 


Further Resources 


e Bahr, E. “Attribution of Biological Weapons 
Use,” in Encyclopedia of Bioterrorism 
Defense. Hoboken, NJ: John Wiley & Sons, 
2005. 


e Mistral Solutions. “Close-In Covert 
Autonomous Disposable Aircraft (CICADA) 
for Homeland Security,” 2014. 


e Piore, A. “Rise of the Insect Drones.” Popular 
Science. January 29, 2014. 





e Gillespie, T., and R. West. “Requirements for 
Autonomous Unmanned Air Systems Set by 
Legal Issues.” International C2 Journal 4, no. 
2 (2010): 1-32. 
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Issue 9: 


The development of AWS will 
lead to a complex and troubling 
landscape of proliferation and 
abuse. 


Background 


Use of AWS by a myriad of actors of different 
kinds, including states (of different types of 
regime) and non-state actors (militia, rebel 
groups, individuals, companies, including private 
military contractors), would lead to such systems 
becoming commonplace anywhere anyone 
favors violence due to the disintermediation and 
scalability afforded by their availability. 


There will be incentives for misuse depending 
upon state of conflict and type of actor. For 
example, such misuse may include, but is not 
limited to, political oppression, crimes against 
humanity, intimidation, assassination, and 
terrorism. This can lead to, for example, a single 
warlord targeting an opposing tribe based on their 
respective interests as declared on Facebook, 
their DNA, their mobile phones, or their 
appearance. 


Candidate Recommendations 


* One must design weapons with high degrees 
of automation in such a way that avoids 
tampering for unintended use. Further work 
on technical means for nonproliferation 
should be explored, for example, 
cryptographic chain authorization. 
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e There is an obligation to consider the 
foreseeable use of the system, and whether 
there is a high risk for misuse. 


e There is an obligation to consider, reflect on, 
or discuss possible ethical consequences of 
one’s research and/or the publication of that 
research. 





Issue 10: 


AWS could be deployed by 
domestic police forces and 
threaten lives and safety. AWS 
could also be deployed for 
private security. Such AWS 
may have very different design 
and safety requirements than 
military AWS. 


Background 


Outside of military uses of AWS, other likely 
applications include use by domestic police 
forces, as well as coast guards, border patrols, 
and other domestic security applications. Police 
in Dallas, Texas used a bomb disposal robot to 
deliver a bomb to kill a suspect in the summer 

of 2016. While that was a remotely operated 
weapon delivered by a remote operated platform, 
the path to more autonomous forms of police 
robots using weapons seems highly likely. 


Beyond use by governments, AWS could 
potentially also be deployed for other private 
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security applications, such as guarding property, 
patrolling areas, and personal protection. 


Tyrants and despots might utilize AWS to gain or 
retain control over a population which would not 
otherwise support them. AWS might be turned 
against peaceful demonstrators when human law 
enforcement might not do the same. 


Candidate Recommendations 


e Police and private security systems should 
not be permitted to deploy weapons without 
meaningful human control. 


e — Police and security systems should deploy 
non-lethal means to disrupt and avert 
security threats and threats to the physical 
safety of humans. 


Further Resources 


e Asaro, P. “Will #BlackLivesMatter to 
RoboCop?” WeRobot 2016, University of 
Miami School of Law, Miami, FL, April 1-2, 
2016. 


e  Asaro, P. "Hands Up, Don't Shoot!’ HRI and 
the Automation of Police Use of Force,” 
Special Issue on Robotics Law and Policy, 
Journal of Human-Robot Interaction 5, no. 3 
(2016): 55-69. 
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Issue 11: 


An automated weapons 

system might not be predictable 
(depending upon its design 

and operational use). Learning 
systems compound the problem 
of predictable use. 


Background 


Autonomous systems that react and adapt to 
environmental and sensor inputs results in 
systems that may be predictable in their general 
behavior, but may manifest individual or specific 
actions that cannot be predicted in advance. 


As autonomous systems become more complex 
in their processing of data, the ability of designers 
to anticipate and predict their behavior becomes 
increasingly difficult. 


As adaptive systems modify their functional 
operations through learning algorithms and other 
means, their behavior becomes more dependent 
upon the content of training data and other 
factors which cannot be anticipated by designers 
or operators. 


Even when a single system is predictable, or even 
deterministic, when such systems interact with 
other systems, or in large masses or swarms, 
their collective behavior can become intrinsically 
unpredictable. This includes unpredictable 
interactions between known systems and 
adversarial systems whose operational behavior 
may be unknown. 
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Modeling and simulation of AWS, particularly 
learning systems, may not capture all possible 
circumstances of use or situational interaction. 
They are underconstrained cyberphysical 
systems. Intrinsic unpredictability of adaptive 
systems is also an issue: one cannot accurately 
model the systems of one’s adversary and how 
an adversary will adapt to your system resulting in 
an inherently unpredictable act. 


Candidate Recommendations 


* Systems that exhibit intrinsically unpredictable 
behavior should be considered illegal and 
not deployed. 


* Similarly, deploying systems with otherwise 
predictable behavior in situations or contexts 
in which the collective behavior of systems 
cannot be predicted should be avoided. In 
particular, deploying AWS swarms in which 
the emergent dynamics of the swarm have 
a significant impact on the actions of an 
individual AWS must be avoided. 


e The predictability of weapons systems 
should be assessed with confidence levels 
with respect to specified contexts and 
circumstances of use. Systems should not be 
used outside of the contexts of use for which 
their operational behavior is understood 
and predictable. Engineers should explicitly 
examine their systems and inform their 
customers of their qualitative and quantitative 
confidence in the predictability of the actions 
of the autonomous functions of weapons 
systems in response to representative 
scenarios, specific contexts of use, and 
scope of operations. 






@greative 
commons 


Commanders and operators should be 
trained to understand and assess confidence 
in the behavior of a system under specific 
contexts and scope of operations. They 
should maintain situational awareness of 
those contexts where weapons systems are 
deployed, and prevent those systems from 
being used outside the scope of operations 
for which their behavior is predictable. 


To ensure meaningful human control, 
operators should be able to query a system 
in real-time. Such a query should offer the 
evidence, explanation, and justification 

for critical determinations made by the 
systems, i.e., identification of a target, 

or key recommendations. 


Weapons systems with advance automation 
should also keep records and traces of critical 
functional and operational decisions that are 
made automatically. Such traces and records 
should be reviewable in instances where the 
behavior of the system was not as predicted. 


To the extent that systems contain adaptive 
or learning algorithms, any critical decision 
made by systems based upon those 
algorithms should be transparent and 
explainable by the designing engineers. 
Any data used for training and adaptation 
should be reviewed as to its integrity so 

as to ensure that learned functions can 
behave in reliably predictable ways. 
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Further Resources 























e — International Committee for Robot Arms * Edwards, S. J. A. Swarming and the Future 
Control. “LAWS: Ten Problems for Global of Warfare, Santa Monica, CA: RAND 
Security” (leaflet). April 10, 2015. Corporation, 2004. 

e Owens, D. "Figuring Forseeability.” Wake e — Rickli, J.-M. “Some Consideration of the 
Forest Law Review 44 (2009): 1277, Impact of Laws on International Security: 
1281-1290. Strategic Stability, Non-State Actors and 

Future Prospects." Meeting of Experts on 

* Scherer, M. “Who's to Blame (Part 5): A Lethal Autonomous Weapons Systems 
Deeper Look at Predicting the Actions of Convention on Certain Conventional 
Autonomous Weapons.” Law and Al, February Weapons (CCW) United Nations Office 
29, 2016. Geneva, April 16, 2015. 

* — Arquilla, J, and D. Ronfeldt. Swarming and * — Scharre, P. Robotics on the Battlefield Part II: 
the Future of Conflict, Santa Monica, CA: The Coming Swarm, Washington, DC: Center 
RAND Corporation, 1997. for a New American Security, 2014. 
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Autonomous and Intelligent systems (A/IS) provide unique and impactful opportunities 

in the humanitarian space. As disruptive technologies, they promise to upend multiple 
historical institutions and corresponding institutional relationships, offering opportunities to 
"re-intermediate” those settings with more humanitarian and equitably focused structures. 


The value of A/IS is significantly associated with the generation of superior and unique 
insights, many of which could help to foster the accomplishment of humanitarian and 
development goals and to achieve positive socio-economic outcomes for both developed 
and developing economies. Among the opportunities for cooperation and collaboration 
at the intersection of A/IS and humanitarian and development issues are the following: 


A/IS have been recognized as key enablers for achieving the goals of humanitarian relief, 
human rights, and the United Nations Sustainable Development Goals. This recognition 
provides the opportunity to demonstrate the positive and supportive role that A/IS can 
play in these critical, but perennially under-resourced and overlooked, areas. 


A/IS are related to, but hold a unique place within, the larger “ICT for development” 
narrative. This intersection creates opportunities for A/IS to be applied in settings where 
commercial and development agendas meet, and to facilitate advances in the administration 
and impact assessment of development programs. 


There is an ongoing narrative on affordable and universal access to communications 
networks and the Internet which invites consideration of how the implementations 
and fruits of A/IS will be made available to populations. 


The narrative of “A/IS for the common good" is starting to present itself in various settings. 
Key elements framing this “common good" conversation relate to the need for it to be 
human-centered and include the need for accountability and to ensure that outcomes 

are fair and inclusive. 
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The scaling and use of A/IS represent a genuine opportunity to provide individuals and 
communities — be they rural, semi-urban, or cities — with greater autonomy and choice. 
A/IS will potentially disrupt all manner of economic, social, and political relationships 
and interactions. Those disruptions will provide a historical opportunity to re-establish 
those settings so that they are reflective of more updated and sustainable notions of 
autonomy and choice. 


Many of the debates surrounding A/IS take place within advanced countries among 
individuals benefiting from adequate finances and higher-than-average living situations. 
It is imperative that all humans in any condition around the world are considered in 
the general development and application of these systems to avoid the risk of bias, 
excessive imbalances, classism, and general non-acceptance of these technologies. 


In the absence of that comprehensive treatment, A/IS policy issues will be addressed 
piecemeal by different jurisdictions and in different sectors. In that context of “distributed 
policy making,” a patchwork of policies and initiatives is the likely result, dissipating potential 
impact. However, some measure of “policy interoperability” can still be served if there is 

a common framing or policy generation process for analysis that can be shared across 
jurisdictions and/or sectors. 


The use of A/IS in support of the pragmatic outcomes noted above is best framed within 
four key domains that comprise the following four sections: economics, privacy and 
safety, education, and equal availability. Each of these contexts presents unique 
challenges, attention to which can inform the trustworthy use of A/IS for the common good. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — Economics 


While the increase of A/IS and its positive uses 

in society are undeniable, the financial gains from 
these technologies may favor certain sectors, and 
are not evenly distributed throughout populations 
where it is created or deployed. Likewise, while 
A/IS automation of certain human tasks may 

be beneficial by supplanting arduous jobs, how 
employment in aggregate for specific populations 
and job verticals will be affected by A/IS needs to 
be addressed. 





Issue: 


A/IS should contribute to 
achieving the UN Sustainable 
Development Goals. 


Background 


The contribution of A/IS to human and 
sustainable development in developing 
countries, and in particular extreme poverty 
eradication, is inherently connected with 

its contribution to human well-being in the 
developed world. In a globalizing society, one 
part of the world has a direct impact on another. 
With a growing level of interdependence 
between communities, the challenges and 
opportunities are truly global. Climate change, 
poverty, globalization, and technology are closely 
interconnected. Ethical commitment should 
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entail a sense of global citizenship and 
of responsibility as members of humanity. 


Beyond considering the humanitarian role of 
A/IS, there is a pressing need to address how 
these technologies can contribute to achieving 
the UN Sustainable Development Goals that 
concern eradicating poverty, illiteracy, gender 
and ethnic inequality, and combating the impact 
of climate change. 


The inequality gap between the developed 

and the developing nations is disturbingly wide. 
With the introduction of hi-tech, the world had 
witnessed a considerable increase in the existing 
gap as the new market is dominated by products 
and services from this new sector. One of the 
factors contributing to this is the nature of the 
tech economy and its tendency to concentrate 
wealth in the hands of few. The tech economy is 
also susceptible to corporate aggregation. 


We need to answer questions such as “How will 
developing nations implement A/IS via existing 
resources? Do the economics of developing 
nations allow for A/IS implementation? What 
should be the role of the public and the private 
sectors and society in designing, developing, 
implementing, and controlling A/IS? How can 
people without technical expertise maintain 
these systems?” 


The risk of unemployment for developing 
countries is more serious than for developed 
countries. The industry of most developing 
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countries is labor intensive. While labor may 
be cheap(er) in developing economies, the 
ripple effects will be felt much more than in 
the developed economies as more and more 
jobs will be gradually replaced along with the 
development of robots or A/IS. 


As an example, in the manufacturing industry, 
lots of products such as mobile phones and 
clothes are designed in developed countries, 
but made in developing countries. Thus, it is not 
difficult to predict that the developing countries 
will be at greater risk of unemployment than 
developed countries if those manufacturing 
tasks can be replaced by machines. The 
challenge of unemployment is even bigger 

for developing countries than for developed 
countries, which can exacerbate the economic 
and power-structure differences between 

and within developed and developing nations. 


Candidate Recommendations 


The current panorama of applications of A/ 

IS in sectors crucial to the UN Sustainable 
Development Goals should be studied, and the 
strengths, weaknesses, and potential of some of 
the most significant recent applications drawn 
from these sectors should be analyzed. Specific 
areas to consider include: 


e Taking appropriate action to mitigate the 
gap. The private sector should integrate 
CSR (corporate social responsibility) at 
the core of development and marketing 
strategies and operations. Mitigating the 
social problems of technology development 
should be a special focus of responsible 
companies using A/S. 
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Developing mechanisms for increasing 
transparency of power structures and justly 
sharing the economic and knowledge 
acquisition benefits of robotics/A/IS. 


Facilitating robotics/A/IS research and 
development in developing nations. 


Empowering the education sector with 
advanced courses on A/S is the first step 
toward creating a nation that can handle the 
new economic and power shifts. 


Investing in technology transfer will help 
developing nations reduce the gap. 


Adapting legal and policy frameworks which 
will help to favor equitable distribution of 
wealth, empowering competent international 
organizations to favor a minimally viable 
competition level on the A/IS markets to 
avoid detrimental monopolistic situations. 


Identifying A/IS technologies relevant to the 
UN Sustainable Development Goals such 
as big data for development (agriculture, 
medical tele-diagnosis), geographic 
information systems (disaster prevention, 
emergency planning), and control systems 
(naturalizing intelligent cities through energy 
and traffic control, management of urban 
agriculture). 


Developing guidelines and 
recommendations for the nurturing and 
implementation of these technologies in 
developing countries. 


Documenting and disseminating successful 
examples of good practice, and evaluations 
and conclusions of experiences. 
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* Developing appropriate impact indices 
for the evaluation of A/IS technological 
interventions in developing countries from 
multiple perspectives. 


Further Resources 


* United Nations. “Sustainable Development 
Goals: 17 Goals to Transform Our World.” 
September 25, 2015. 











Issue: 


It is unclear how developing 
nations can best implement 
A/IS via existing resources. 


Background 


Do the economics of developing nations allow 
for A/IS implementation? How can people 
without technical expertise maintain design 
specifications and procure these systems? 

The potential use of A/IS to create sustainable 
economic growth for LMICs (lower and middle 
income countries) is uniquely powerful. If 

A/IS capacity and governance problems are 
addressed, LMICs will have the ability to use 
A/IS to transform their economies and leapfrog 
into a new era of inclusive growth if a clear path 
for development is provided. Particular attention, 
however, should be paid to ensure that the use 
of A/IS for the common good — especially in the 
context of LMICs — does not reinforce existing 
socio-economic inequities. 
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Candidate Recommendations 


Develop mechanisms for increasing 
transparency of power structures and justly 
sharing the economic and knowledge 
acquisition benefits of A/IS. 


Facilitate A/IS research and development 
in developing nations. Ensure that 
representatives of developing nations 
are involved. 


Along with the use of A/IS, discussions 
related to identity, platforms, and 
blockchain are needed to ensure that 
all of the core enabling technologies are 
designed to meet the needs of LMICs. 


Further Resources 


Ajakatye, O., and M. S. Kimenyi. “Higher 
Education and Economic Development in 
Africa: Introduction and Overview." Journal of 
African Economies 20, no. 3 (2011): iii3—iii13. 


Bloom, D. E., D. Canning, and K. Chan. Higher 
Education and Economic Development in 
Africa (Vol. 102). Washington, DC: World 
Bank, 2006. 


Bloom, N. “Corporations in the Age of 
Inequality.” Harvard Business Review, April 
21, 2017. 





Dahlman, C. Technology, Globalization, and 
Competitiveness: Challenges for Developing 
Countries. Industrialization in the 21st 
Century. New York: United Nations, 2006. 
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+ Fong, M. Technology Leapfrogging for 
Developing Countries. Encyclopedia of 
Information Science and Technology, 2nd ed. 
Hershey, PA: IGI Global, 2009 (pp. 3707- 
3713). 


e Frey, C. B., and M. A. Osborne. “The Future 
of Employment: How Susceptible Are Jobs to 
Computerisation?” (working paper). Oxford, 
U.K.: Oxford University, 2013. 








e — Rotman, D. “How Technology Is Destroying 
Jobs.” MIT Technology Review, June 12, 2013. 


* McKinsey Global Institute. “Disruptive 
Technologies: Advances That Will Transform 
Life, Business, and the Global Economy” 
(report), May 2013. 


* Sauter, R., and J. Watson. “Technology 
Leapfrogging: A Review of the Evidence, 
A Report for DFID.” Brighton, England: 
University of Sussex. October 3, 2008. 


e “The Rich and the Rest.” The Economist. 
October 13, 2012. 





e “Wealth Without Workers, Workers Without 
Wealth.” The Economist. October 4, 2014. 





e World Bank. “Global Economic Prospects 
2008: Technology Diffusion in the 
Developing World.” Washington, DC: World 
Bank, 2008. 
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Issue: 


The complexities of 
employment are being 
neglected regarding A/IS. 


Background 


Current attention on automation and 
employment tends to focus on the sheer number 
of jobs lost or gained. Other concerns include 
changes in traditional employment structures. 


Candidate Recommendations 


It is important to focus the analysis on how 
employment structures will be changed by 
automation and Al rather than on solely dwelling 
on the number of jobs that might be impacted. 
The analysis should focus on how current 

task content of jobs are changed based on a 
clear assessment of the automatibility of the 
occupational description of such jobs. 


While there is evidence that robots and 
automation are taking jobs away in various 
sectors, a more balanced, granular, analytical, 
and objective treatment of this subject will more 
effectively help inform policy making. Specifics to 
accomplish this include: 


e Create an international, independent agency 
which can properly disseminate objective 
Statistics and inform media as well as the 
general public about the impact of robotics 
and A/IS on jobs and growth. 
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* Consider both product and process 
innovation and look at it from a global 
perspective as a way to understand properly 
the global impact of A/IS on employment 
(refer to Pianta, 2009 and Vivarelli 2007). 


e Focus the analysis on how employment 
structures will be changed by A/IS rather 
than on the number of jobs that might 
be impacted. The analysis should focus 
on how current task-content of jobs are 
changed based on a clear assessment of the 
automatibility of the occupational description 
of such jobs (refer to Bonin et al. 2015 and 
RockEU, 2016). 


e Make sure workers can improve their 
adaptability to fast technological changes by 
providing them adequate training programs. 
Those training programs could be available 
to any worker with a special attention to 
low-skilled workforce members. Those 
programs can be private (sponsored by the 
employer) or public (offered freely through 
specific public channels and policies), and 
they should be open while the worker is in- 
between jobs or still employed. 


e — Ensure that not only the worker whose job is 
concerned benefits from training programs, 
but also any employee in the company so 
everyone has the chance to be up to speed 
with technical changes, even if one’s job 
is not immediately concerned (not only 
reaction but also prevention). Thus it should 
be the responsibility of every company to 
increase its investment in the internal training 
of its workforce based on the profitability 
gains realized thanks to automation. The 
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public side could facilitate such initiatives 
with co-investment in the training programs 
through tax incentives. 


Further Resources 


* RockEU. “Robotics Coordination Action 
for Europe Report on Robotics and 
Employment,” Deliverable D3.4.1, June 30, 
2016. 


e — International Federation of Robotics. 
“The Impact of Robots on Productivity, 
Employment and Jobs,” A positioning paper 
by the International Federation of Robotics, 
April 2017. 








*  Brynjolfsson, E., and A. McAfee. The 
Second Age of Machine Intelligence: Work 
Progress and Prosperity in a Time of Brilliant 
Technologies. New York: W. W. Norton & 
Company, 2014. 





Issue: 


Automation is often viewed 
only within market contexts. 


Background 


A/IS are expected to have an impact beyond 
market domains and business models. Examples 
of impact include safety, public health, and 
socio-political considerations of deploying A/ 

IS. This impact will diffuse through the global 
society. 
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Candidate Recommendation 


To understand the impact of A/IS on society, it is 
necessary to consider both product and process 
innovation as well as wider implications from a 
global perspective. 


Further Resources 


e Pianta, M. Innovation and Employment, 
Handbook of Innovation. Oxford, U.K.: 
Oxford University Press, 2003. 


e Vivarelli, M. “Innovation and Employment: A 
Survey,” Institute for the Study of Labor (IZA) 
Discussion Paper No. 2621, February 2007. 





Issue: 


Technological change 

is happening too fast for 
existing methods of 
(re)training the workforce. 


Background 


The current pace of technological development 
will heavily influence changes in employment 
structure. In order to properly prepare the 
workforce for such evolution, actions should be 
proactive and not only reactive. 


The wave of automation caused by the A/IS 
revolution will displace a very large amount of 
jobs across domains and value chains. The U.S. 
"automated vehicle” case study analyzed in the 
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White House 2016 report Artificial Intelligence, 
Automation, and the Economy is emblematic 
of what's at stake: 2.2 to 3.1 million existing 
part- and full-time U.S. jobs are exposed over 
the next two decades, although the timeline 
remains uncertain. In particular, between 1.3 
and 1.7 million heavy truck drivers are 
threatened. And this is not trivial, for the 
profession has symbolized in the collective 
imagination the manifestation of the American 
dream of empowerment, liberty, and social 
ascension whereby less-educated people could 
make it into the middle class. 


The automation wave calls at least for higher 
investment and probably the need to reinvent 
active labor market programs in the coming 
decades. Such investment should logically be 
funded by fiscal policies targeting the capital. 
The 2016 White House report gave an interesting 
order of magnitude applied to the case of the 
United States: “increasing funding for job training 
in the U.S. by six-fold — which would match 
spending as a percentage of GDP to Germany, 
but still leave the U.S. far behind other European 
countries — would enable retraining of an 
additional 2.5 million people per year." 


A/IS and other digital technologies offer real 
potential to innovate new approaches to 
job-search assistance, placement, and hiring 
processes in the age of personalized services. 
The efficiency of matching labor supply and 
demand can be tremendously enhanced by 

the rise of multi-sided platforms and predictive 
analytics. The case of platforms, such as LinkedIn 
for instance with its 470 million registered users, 
is interesting as an evolution in hiring practices. 
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Tailored counseling and integrated re-training 
programs also represent promising grounds for 
innovation. 


This, however, will not be enough. A lot will have 
to be done to create fair and effective life-long 
skill development/training infrastructure and 
mechanisms capable of empowering millions 

of people to viably transition jobs, sectors, and 
potentially geographies. A lot will also have to be 
done to address differential geographic impacts 
which exacerbate income and wealth disparities. 
Effectively enabling the workforce to be more 
mobile — physically, legally, and virtually — will be 
crucial. This implies systemic policy approaches 
which encompass housing, transportation, 
licensing, taxes, and, crucially in the age of A/IS, 
broadband access, especially in rural areas. 


Candidate Recommendations 


e To cope with the technological pace and 
ensuing progress of A/IS, it will be necessary 
for workers to improve their adaptability 
to rapid technological changes through 
adequate training programs provided to 
develop appropriate skillsets. Training 
programs should be available to any worker 
with special attention to the low-skilled 
workforce. Those programs can be private 
(sponsored by the employer) or public 
(offered freely through specific public 
channels and policies), and they should be 
open while the worker is in between jobs or 
still employed. Fallback strategies also need 
to be developed for those who cannot be 
re-trained. 
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To lay solid foundations for the profound 
transformation outlined above, more research 
in at least three complementary areas 

is needed: 


e First, to devise mechanisms of 
dynamic mapping of tasks and 
occupations at risks of automation 
and associated employment 
volumes. This mapping of the 
workforce supply is needed at the 
macro, but also crucially at the micro, 
levels where labor market programs 
are deployed; 


* Integrated with that, more granular 
and dynamic mapping of the future 
jobs/tasks, workplace-structures, 
associated work-habits, and skill- 
base spurred by the A/IS revolution 
are also needed. This mapping of 
the demand side will be key to 
innovate, align, and synchronize skill 
development and training programs 
with future requirements. 


e More policy research on the 
dynamics of professional transitions 
in different labor market conditions 
is required. 


To maximize intended impact, create 
necessary space for trial-and-error strategies, 
and to scale up solutions that work, 
implement robust, data-driven evidence- 
based approaches. These approaches should 
be based on experiments and centered on 
outcomes in terms of employment but also 
in terms of earnings. New forms of people- 
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public-private partnerships involving civil Further Resources 
society as well as new outcome-oriented 
financial mechanisms (social impact bonds, 
for instance) that help scale up successful 
innovations should also be considered. 


e Executive Office of the President. Artificial 
Intelligence, Automation, and the Economy. 
December 20, 2016. 





e  Kilcarr, S. “Defining the American Dream for 
Trucking ... and the Nation, Too,” FleetOwner, 





e The next generation of highly qualified 
personnel should be ready to close skills 








gaps and develop future workforces. New paste 

programs should be offered possibly earlier * OECD, “Labor Market Programs: Expenditure 
than high school, to increase access to and Participants,” OECD Employment and 
employment in the future. Labor Market Statistics (database), 2016. 
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Section 2 — Privacy and Safety 


The growing volumes of private sector data 
(mobile phones, financial transactions, retail, 
logistics) hold unique promise in developing 
more robust and actionable disease-monitoring 
systems that can be empowered by A/IS. 
However, concerns related to privacy, the ability 
of individuals to opt out, the cross-border nature 
of data flows, and the political and commercial 
power dynamics of this data are the key factors 
to consider in how to most equitably shape 

this domain. 





There is a lack of access 
and understanding regarding 
personal information. 


Background 


How to handle privacy and safety issues, 
especially as they apply to data in humanitarian 
and development contexts? Urgent issues around 
individual consent, potential privacy breaches, 


and potential for harm or discrimination regarding 


individual's personal data require attention and 
standardized approaches. 


This is especially true with populations that are 
recently online, or lacking a good understanding 
of data use and the ambiguities of data 
“ownership,” privacy, and how their digital 
access generates personal data by-products 
used by third parties. 


According to the GSMA, the number of mobile 
Internet users in the developing world will 
double from 1.5 billion in 2013 to 3 billion 

by 2020, rising from 25% of the developing 
world population to 45% over the period. In 
Sub-Saharan Africa, just 17% of the population 
were mobile Internet subscribers in 2013, but 
penetration is forecast to increase to 37% by 
2020—making the generation, storage, use, 
and sharing of personal data in the developing 
world an issue that will continue to gain gravity. 





In the humanitarian sector, digital technologies 
have streamlined data collection and data 
sharing, frequently enabling improved outcomes. 
With a focus on rights and dignity of the 
populations served, practitioners and agencies 
have advocated for more data sharing and open 
data in the social good sector. Timely access to 
public, social sector, and private data will speed 
response, avoid collection duplications, and 
provide a more comprehensive summary of a 
situation, based on multiple data streams and a 
wider range of indicators. 


However, there are inherent risks when multiple 
sources of data are overlaid and combined to 
gain insights, as vulnerable groups or individuals 
can be inadvertently identified in the process. 
The privacy threat is the most discussed risk: 
When is informed consent or opt-in really 
ethical and effective? Best practices remain an 
unresolved issue among practitioners when 
working with communities with fewer resources, 
low literacy, lower connectivity, and less 
understanding about digital privacy. 
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The “do no harm” principle is practiced in 
emergency and conflict situations. Humanitarian 
responders have a responsibility to educate 
the populations about what will happen with 
their data in general, and what might happen if 
it is shared openly; there is often lack of clarity 
around how these decisions are currently being 
made and by whom. Remedial steps should 
include community education regarding digital 
privacy, as well as helping vulnerable groups 
become more savvy digital citizens. 


There are perception gaps regarding what 
constitutes potential and actual harm stemming 
from data use practices. A collaborative 
consensus across sectors is needed on 
safeguarding against risks in data collection, 
sharing, and analysis — particularly of combined 
sets. From the outset, iterative, ethics-based 
approaches addressing data risk and privacy are 
key to identify and mitigate risks, informing better 
action and decision-making in the process. 


Candidate Recommendation 


Frameworks such as Privacy by Design can guide 
the process of identifying appropriate system 
and software requirements in early stages 

of design. Such frameworks also encourage 
proactive examination of harms and risks, seek 
to engage the data subject (e.g., consumer, user, 
stakeholders) in the design of the software, 

and recommend best practices and regulatory 
requirements (such as data minimization, 
accountability, transparency, options such as 
opt-in, opt-out, encryption) to be embedded 
into the system. /n addition: 


e Best practices such as Privacy Impact 
Assessments will assist with identification of 
data misuse cases at early stages of system/ 
software design. 


* — Improving digital literacy of citizens should be 
a high priority for the government and other 
organizations. 


e Governments should enforce transparency 
related to data collection, data ownership, 
data stewardship, and data usage and 
disclosure. 


* Organizations should be held accountable 
for data misuse, financial loss, and harm to 
the reputation of the data object if data is 
mishandled. This requires that organizations 
have appropriate policies and agreements 
in place, that terms and conditions of the 
agreements are clearly communicated with 
the data object and that data misuse cases 
and legitimate use cases are well-defined 
in advance. 


Further Resources 


e For more on responsible data use, please 
see the section “Personal Data and Individual 
Access Control.” 


e For more on responsible data use, see 
the Responsible Development Data Book. 
Oxfam also has a responsible data policy that 
provides a field-tested reference. 


e Example Use Case from GSMA: When Call 
Data Records (CDRs) are used to help in 
the response to the Ebola outbreak, mobile 
operators wish to ensure mobile users’ 
privacy is respected and protected and 
associated risks are addressed. 








e van Rest J., D. Boonstra, M. Everts, M. van 
Rijn, R. van Paassen. “Designing Privacy-by- 
Design,” in Privacy Technologies and Policy, 
edited by B. Preneel, and D. Ikonomou. 
Lecture Notes in Computer Science, vol. 
8319. Berlin, Germany: Springer, 2012. 
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Section 3 — Education 


It is essential to increase the awareness, 
critical understanding, and attitudinal values 

of undergraduate and postgraduate students 
related to sustainable human development 
and its relationship with A/IS, so that they are 
prepared to assume their responsibilities in 
the solution of the current global social crises. 
Current and future leaders should be educated 
in macro-ethics and not only in micro-ethics. 


Shared narratives, generated by awareness, 
education, and standard evaluative models 
are the best pathway to generating the global 
support necessary to meet these challenges. 
Programs fostering awareness, education, 
and analytical and governance models should 
address the opportunities and risks of A/IS 

in development contexts. 





Issue: 


How best to incorporate the 
“global dimension of engineering” 
approach in undergraduate and 
postgraduate education in A/IS. 


Background 


A/IS presents a unique opportunity for 
narrative and policy construction in educational 
institutions. Where norms exist, they are taught 
in schools. Thus, physics majors learn the 
“standard” theories and equations of physics. 
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The same is true in other disciplines. However, 
where standards are either absent or in the 
process of development in a sector, what is 
most appropriately included in undergraduate 
and graduate curriculum is less clear. That is the 
case for a number of areas in the digital world, 
including A/IS. Thus, educators and other parties 
involved in curriculum development should 
consider the opportunity to craft curricula that 
will make their students aware of this absence 
of standards, and also encourage the exploration 
of various practices as candidates for “best 
practices” and their possible further elevation to 
standards in Al technology and policy. 


Candidate Recommendations 


The understanding of the global dimension 
of engineering practice should be embedded in 
A/IS curricula. Specifically: 


* Curriculum/core competencies should 
be defined and preparation of course- 
material repositories and choice of the most 
adequate pedagogical approaches should 
be established. 


e The potential of A/IS applications should 
be emphasized in undergraduate and 
graduate programs specifically aimed at 
engineering in international development 
and humanitarian relief contexts as well as 
in the training programs preparing technical 
professionals for work in the international 
development and humanitarian sectors. 
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e Increased awareness on the opportunities Further Resource 
and risks faced by Lower Middle Income 
Countries (LMICs) in the use of A/IS for 
sustainable development and humanitarian 
purposes is critical. Ignoring these 
opportunities and risks will further divide 
the opportunities for development across 
the globe. A/IS presents an opportunity to 
potentially reduce these differentials that 
ultimately strain social fabric and economic 
systems. 


e Global Dimension in Engineering Education 


Project (GDEE). 
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Section 4 — Equal Availability 


For A/IS to be adopted in an atmosphere of trust 
and safety, greater efforts must be undertaken to 
increase availability of these resources. 





Issue: 


Al and autonomous technologies 
are not equally available 
worldwide. 


Background 


Equitable distribution of the benefits of A/IS 
technology worldwide should be prioritized. 
Training, education, and opportunities in A/IS 
worldwide should be provided particularly with 
respect to underdeveloped nations. 


Candidate Recommendations 


Working with appropriate organizations 

(e.g., the United Nations) stakeholders from 

a cross-sectional combination of government, 
corporate, and non-governmental organization 
(NGO) communities should: 


1. Engage in discussions regarding effective 
A/IS education and training. 


2. Encourage global standardization/ 
harmonization and open source software 
for A/IS. 
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3: 


Promote distribution of knowledge and 
wealth generated by the latest A/IS, 
including formal financial mechanisms 
(such as taxation or donations to effect 
such equity worldwide). 


International organizations, government 
bodies, universities, and research 
institutes should promote research into 
A/IS technologies that are readily available 
in developing countries, for example, 
mobile lightweight A/IS applications 
(taking advantage of the widespread use 
of increasingly affordable Internet-enabled 
phones in developing contexts) and 
culture-aware systems. 


National and international development 
cooperation agencies and NGOs should 
draw attention to the potential role of A/IS 
in human and sustainable development. 


Further Resources 


Hazeltine, B., and C. Bull. Appropriate 
Technology: Tools, Choices, and Implications. 
New York: Academic Press, 1999. 


Akubue, A. “Appropriate Technology for 
Socioeconomic Development in Third World 
Countries.” The Journal of Technology 
Studies 26, no. 1 (2000): 33—43. 
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The first edition of the law section for Ethically Aligned Design noted that the early stages 
in development of autonomous and intelligent systems (A/IS) have given rise to many 
complex ethical problems that translate directly and indirectly into discrete legal challenges. 
That is, of course, what the rule of law often intends to answer — how we should behave 
as a society when faced with difficult ethical decisions — and it should come as no surprise 
that the legal implications of A/IS continue to unfold as we witness the forms of its 
expression and use expand. 


To consider the ongoing creep of A/IS ethical issues into the legal realm, one need look 
no further than the first section of this document: Legal Status. This section addresses 
what legal status should A/IS be granted and was not a topic in the original edition. That 
is to say, in just one revision of this paper, we felt the need to address the question of 
how A/IS should be labeled in the courts’ eyes: a product that can be bought and sold? 
A domesticated animal with more rights than a simple piece of property, but less than a 
human? A person? Something new? 


Our conclusion to that question is that A/IS are not yet deserving of any kind of 
“personhood” — yet the very fact that the question of whether A/IS could, or should, be 
granted such status demonstrates the rate at which the technology and the related legal and 
ethical questions are growing and provide two universal principles echoed throughout this 
document: 


The development, design, and distribution of A/IS should fully comply with all applicable 
international and domestic law. 


There is much work to be done: the legal and academic community must increase 
engagement in this rapidly developing field from its members. 
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Concerns and recommendations fall into four main areas: 
1. Legal Status of A/IS 





2. Governmental Use of A/IS: Transparency and Individual Rights 





3. Legal Accountability for Harm Caused by A/IS 








4. Transparency, Accountability, and Verifiability in A/IS 


While much debate continues to surround A/IS, its development, and use, these questions 
must be addressed before the proliferation of A/IS passes some kind of tipping point. 

The authors hope this paper will inform the legislative process and inspire more members 
of the legal community to become involved now. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 





@greative 
commons 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 147 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 





Section 1 — Legal Status of A/IS 


There has been much discussion about how to 
legally regulate A/IS-related technologies, and 
the appropriate legal treatment of systems that 
deploy these technologies. Lawmakers today 
are wrestling with the issue of what status 

to apply to A/IS. Legal “personhood” (as is 
applied to humans and certain types of human 
organizations) is one possible option for 
framing such legal treatment, and the 
implications of granting that status to A/IS 
applications raises issues that have implications 
in multiple domains of human interaction 
beyond technical issues. 





What type of legal status (or 
other legal analytical framework) 
is appropriate for application 

to A/IS, given the legal issues 
raised by deployment of such 
technologies? 


Background 


The convergence of A/IS and robotics 
technologies has led to the development 

of systems and devices with attributes that 
resemble those of human beings in terms of their 
autonomy, ability to perform intellectual tasks 
and, in the case of some robots, their physical 
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appearance. As some types of A/IS begin to 
display characteristics that resemble those of 
human actors, some governmental entities and 
private commentators have concluded that it 

is time to examine how legal regimes should 
categorize and treat various types of A/IS. These 
entities have posited questions such as, “Should 
the law treat such systems as legal ‘persons,’ with 
all the rights and responsibilities that personhood 
entails?” Such status seems initially remarkable 
until consideration is given to the long-standing 
legal personhood status granted to corporations, 
governmental entities, and the like — none 

of which are human even though they are 

run by humans. 





Alternatively, many entities have asked, should 
some A/IS be treated as mere products and tools 
of their human developers and users? Perhaps 
A/IS are something entirely without precedent, 
raising the question of whether one or more 
types of A/IS might be assigned an intermediate 
— and perhaps novel — type of legal status? 


Clarifying the legal status of A/IS in one or 
more jurisdictions is essential in removing the 
uncertainty associated with the obligations and 
expectations for organization and operation of 
these systems. Clarification along these lines 
will encourage more certain development and 
deployment of A/IS and will help clarify lines of 
legal responsibility and liability when A/IS cause 
harm. Recognizing A/IS as independent “legal 
persons” would, for example, limit or eliminate 
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some human responsibility for subsequent 
“decisions” made by such A/IS (for example 
under a theory of “intervening causation” — akin 
to the “relief” from responsibility of a hammer 
manufacturer when a burglar uses a hammer to 
break the window of a house), thus potentially 
reducing the incentives for designers, developers, 
and users of A/IS to ensure their safety. In 

this example, legal issues that are applied in 
similar “chain of causation” settings (such as 
“foreseeability,” “complicity,” “reasonable care,” 
“strict liability” for unreasonably dangerous goods, 
and other precedential notions) will factor into 
the design process. Different jurisdictions may 
reach different conclusions about the nature of 
such causation chains, inviting future creative 
legal planners to consider how and where to 
pursue design, development, and deployment of 
future A/IS in order to receive the most beneficial 
legal treatment. 





"n 





The issue of the legal status of A/IS thus 
intertwines with broader legal questions regarding 
how to ensure accountability and assign and 
allocate liability when A/IS cause harm. The 
question of legal personhood for A/IS also 
interacts with broader ethical questions on the 
extent to which A/IS should be treated as moral 
agents independent from their human designers 
and operators, and whether recognition of A/IS 
personhood would enhance or detract from the 
purposes for which humans created the A/IS in 
the first place. 


A/IS are at an early stage of development where 
it is premature to assert a single particular legal 
status or presumption for application in the many 
forms and settings in which those systems are 


deployed. This uncertainty, coupled with the 
multiple legal jurisdictions in which A/IS are being 
deployed (each of which, as a sovereign, can 
regulate A/IS as it sees fit) suggests that there 

are multiple general frameworks through which 
to consider A/IS legal status. Below are some 
examples. 


Candidate Recommendations 


1. While conferring legal personhood on A/ 
IS might bring some economic benefits, 
the technology has not yet developed 
to the point where it would be legally or 
morally appropriate to generally accord A/ 
IS the rights and responsibilities inherent in 
the legal definition of personhood, as it is 
defined today. Therefore, even absent the 
consideration of any negative ramifications 
from personhood status, it would be unwise 
to accord such status to A/IS at this time. A/ 
IS should therefore remain to be subject to 
the applicable regimes of property law. 


2. Government and industry stakeholders alike 
should identify the types of decisions and 
operations that should never be delegated 
to A/IS, and adopt rules and standards that 
ensure effective human control over those 
decisions. Modern legal systems already 
address a number of other situations that 
could serve as appropriate analogues for the 
legal status of A/IS and how to allocate legal 
responsibility for harm caused by A/IS. These 
legal analogues may include the treatment of 
pets, livestock, wild animals, employees, and 
other “agents” of persons and corporations. 
Governments and courts should review 









@greative 
commons 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 149 





The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 





these various potential legal models and 
assess whether they could serve as a proper 
basis for assigning and apportioning legal 
rights and responsibilities with respect to the 
deployment and use of A/IS. 


3. In addition, governments should scrutinize 
existing laws — especially those governing 
business organizations — for mechanisms 
that could have the practical effect of 
allowing A/IS to have legal autonomy. If 
ambiguities or loopholes in the law could 
create a legal method for recognizing A/IS 
personhood, the government should review 


and, if appropriate, amend the pertinent laws. 


4. Manufacturers and operators should gain 
an understanding of how each jurisdiction 
would categorize a given A/IS and how each 
jurisdiction would treat harm caused by the 
system. Manufacturers and operators should 
be required to comply with the applicable 
laws of all jurisdictions in which that system 
could operate. In addition, manufacturers and 
operators should be aware of standards of 
performance and measurement promulgated 
by standards development organization and 
agencies. 


5. As A/IS become more sophisticated, 
governments should reassess the issue 
of legal status for these systems. In 
considering whether to accord legal 
protections, rights, and responsibilities to 
A/IS, governments should exercise utmost 
caution. Governments and decision-makers 
at every level must work closely with 
regulators, representatives of civil society, 
industry actors, and other stakeholders to 
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ensure that the interest of humanity — and 
not the interests of the autonomous systems 
themselves — remains the guiding principle. 
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Section 2 — Governmental Use of A/IS: 
Transparency and Individual Rights 


Surveillance of populations by governments 

and the disruption of free elections will become 
ever easier as we deploy A/IS. How should we 
manage these systems to ensure that they act for 
the good of society? 





Issue: 


International, national, and local 
governments are using A/IS. 
How can we ensure the A/IS 
that governments employ do 
not infringe on citizens’ rights? 


Background 


Government increasingly automates part or all of 
its decision-making. Law mandates transparency, 
participation, and accuracy in government 
decision-making. When government deprives 
individuals of fundamental rights, individuals are 
owed notice and a chance to be heard to contest 
those decisions. A key concern is how legal 
commitments of transparency, participation, and 
accuracy can be guaranteed when algorithmic- 
based A/IS make important decisions about 
individuals. 
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1. 


Candidate Recommendations 


Government stakeholders should identify 
the types of decisions and operations that 
should never be delegated to A/IS, such as 
when to use lethal force, and adopt rules 
and standards that ensure effective human 
control over those decisions. 


Governments should not employ A/IS that 
cannot provide an account of the law and 
facts essential to decisions or risk scores. 
The determination of, for example, fraud by 
a citizen should not be done by statistical 
analysis alone. Common sense in the A/IS 
and an ability to explain its logical reasoning 
must be required. Given the current abilities 
of A/IS, under no circumstances should court 
decisions be made by such systems alone. 
Parties, their lawyers, and courts must have 
reasonable access to all data and information 
generated and used by A/IS technologies 
employed by governments and other 

state authorities. 


A/IS should be designed with transparency 
and accountability as primary objectives. 
The logic and rules embedded in the system 
must be available to overseers of systems, 

if possible. If, however, the system's logic 

or algorithm cannot be made available for 
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inspection, then alternative ways must 
be available to uphold the values 

of transparency. Such systems should 
be subject to risk assessments and 
rigorous testing. 


4. Individuals should be provided a forum to 
make a case for extenuating circumstances 
that the A/IS may not appreciate — in other 
words, a recourse to a human appeal. 
Policy should not be automated if it has not 
undergone formal or informal rulemaking 
procedures, such as interpretative rules and 
policy statements. 


5. Automated systems should generate audit 
trails recording the facts and law supporting 
decisions and such systems should be 
amenable to third-party verification to 
show that the trails reflect what the system 
in fact did. Audit trails should include a 
comprehensive history of decisions made in 
a case, including the identity of individuals 
who recorded the facts and their assessment 
of those facts. Audit trails should detail 
the rules applied in every mini-decision 
made by the system. Providers of A/IS, 
or providers of solutions or services that 
substantially incorporate such systems, 
should make available statistically sound 
evaluation protocols through which they 
measure, quality assure, and substantiate 
their claims of performance, for example, 
relying where available on protocols and 
standards developed by the National Institute 
of Standards and Technology (NIST) or other 
standard-setting bodies. 
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6: 


Investor list(s), developers, and promoters of 
any given A/IS being developed should be 
required by law to be made public when the 
A/IS are used for governmental purposes. 
There should also be transparency of the 
specific ethical values promoted by the 
designer, and how they were embedded in 
the system. Transparency should also apply 
to the input data selection process. 
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Section 3 — Legal Accountability for 


Harm Caused by A/IS 


As A/IS becomes more prevalent while also 
potentially becoming more removed from the 
human developer/manufacturer, what is the 
correct approach to ensure legal accountability 
for harms caused by A/IS? 





Issue: 


How can A/IS be designed to 
guarantee legal accountability for 
harms caused by these systems? 


Background 


One of the fundamental assumptions most laws 
and regulations rely on is that human beings are 
the ultimate decision-makers. As autonomous 
devices and A/IS become more sophisticated and 
ubiquitous, that will increasingly be less true. The 
A/IS industry legal counsel should work with legal 
experts to identify the regulations and laws that 
will not function properly when the “decision- 
maker” is a machine and not a person. 


Candidate Recommendations 


Any or all of the following can be chosen. The 
intent here is to provide as many options as 
possible for a way forward for this principle. 
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1. Designers should consider adopting an identity 


tag standard — that is, no A/IS agent should 
be released without an identity tag to maintain 
a clear line of legal accountability. 


2. Lawmakers and enforcers need to ensure that 


the implementation of A/IS is not abused 

by businesses and entities employing the A/ 
IS to avoid liability or payment of damages. 
Governments should consider adopting 
regulations requiring insurance or other 
guarantees of financial responsibility so that 
victims can recover damages for harm that A/ 
IS cause. 


3. Companies that use and manufacture A/ 


IS should be required to establish written 
policies governing how the A/IS should be 
used, including the real-world applications 
for such Al, any preconditions for its effective 
use, who is qualified to use it, what training 
is required for operators, how to measure the 
performance of the A/IS, and what operators 
and other people can expect from the A/IS. 
This will help to give the human operators 
and beneficiaries an accurate idea of what to 
expect from the A/IS, while also protecting 
the companies that make the A/IS from future 
litigation. 
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4. Because the person who activates the A/IS 


will not always be the person who manages 
or oversees the A/IS while it operates, states 
should avoid adopting universal rules that 
assign legal responsibility and liability to the 
person who “turns on” the A/IS. For example, 
liability may attach to the manufacturers or to 
the person who directs, monitors, and controls 
the A/IS's operations, or has the responsibility 
to do so. 


6. For the avoidance of repeated or future harm, 


companies that use and manufacture A/IS 
should consider the importance of continued 
algorithm maintenance. Maintenance is an 
essential aspect of design. Design does not 
end with deployment. Thus, there should 

be a clear legal requirement of (1) due 
diligence, and (2) sufficient investment 

in algorithm maintenance on the part of 
companies that use and manufacture 

A/IS that includes monitoring of outcomes, 
complaint mechanism, inspection, correction, 
and replacement of harm-inducing algorithm, 
if warranted. Companies should be prohibited 
from contractually delegating this responsibility 
to unsophisticated end-users. 


7. Promote international harmonization of 


national legislations related to liability in the 
context of A/IS design and operation (through 
bi- or multilateral agreements) to enhance 
interoperability, and facilitate transnational 
dispute resolution. 


8. Courts weighing A/IS litigation cases based 


on some form of injury should adopt a similar 
scheme to that of product liability litigation, 





wherein companies are not penalized or held 


responsible for installing post-harm fixes on 
their products designed to make the product 
safer. In other words, because courts have 
recognized that it is good public policy to 
encourage companies to fix dangerous design 
flaws, retroactively fixing a design flaw that has 
caused injury is not considered an admission 
or a sign of culpability. The same approach 
should be used in A/S litigation. 
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Section 4 — Transparency, Accountability, 
and Verifiability in A/IS 


Transparency around A/IS is a difficult issue 
because it impinges on the differing needs of 
developers for trade secrets and users to be 
able to understand the technology to guard 
against problems occurring with it, and to hold 
accountable the correct entity in the event of a 
system failure. 





Issue: 


How can we improve the 
accountability and verifiability 
in autonomous and intelligent 
systems? 


Background 


Decision-making algorithms can be designed 

for various purposes, and the applications 

are wide-ranging for both the public and the 
private sectors. We must assume that virtually 
every decision that we make as humans can be 
mediated or replaced by an algorithm. Therefore, 
we cannot overestimate both the current and 
future role of A/IS across different sectors. 
Algorithms and automated decision-making (e.g., 
resume/cv screening during job applications) 


have the potential to be more fair, and less 
biased than humans, provided that the systems 
are designed well. This requires, in particular, that 
effective preventative measures are put in place 
to avoid an algorithm-based information and/or 
value bias. 


At the same time, most users of A/IS will not 

be aware of the sources, scale, varying levels of 
accuracy, intended purposes, and significance of 
uncertainty in the operations of A/IS, or that they 
are interacting with A/IS in the first place. The 
sources of data used to perform these tasks are 
also often unclear. Furthermore, users might not 
foresee the inferences that can be made about 
them or the consequences when A/S are used. 
The proliferation of A/IS will result in an increase 
in the number of systems that rely on machine 
learning and other developmental systems whose 
actions are not pre-programmed, and that may 
not produce logs or a record of how the system 
reached its current state. 


These systems are often opaque (frequently 
referred to as “black boxes”) and create 
difficulties for everyone, from the engineer, 

to the lawyer in court, to the online shopper, to 
the social media user. The result is an abundance 
of ethical issues of ultimate accountability. 
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Candidate Recommendations 


1. 


@greative 
commons 


Given that many of the desired design 
specifications regarding accountability and 
verifiability are not technologically possible 

at this time, for now, this is an ethical issue 
that is best addressed by disclosure. If users 
are aware that they are interacting with an A/ 
IS in the first place, and know exactly what 
information is being transferred to it, they 
will be better suited to tailor their inputs. A 
government-approved labeling system like 
the skull and crossbones found on household 
cleaning supplies that contain poisonous 
compounds could be used for this purpose 
to improve the chances that users are aware 
when they are interacting with A/IS. 


Designers and manufacturers must remain 
accountable for the risks or externalities 
their systems cause. This is a balancing 

act since the level of risk that is acceptably 
mitigated through disclosure is not always 
clear. Recommending specific levels (whether 
a manufacturer of A/IS acts responsibly, 

or whether there is enough disclosure, or 
whether total disclosure would even be 
enough to mitigate the risk to users) is often 
a fact-specific discussion that doesn't suit 
itself well to broad rules. 


There is a demand for algorithmic operation 
transparency. Although it is acknowledged 

this cannot be done currently, A/IS should be 
designed so that they always are able, when 
asked, to show the registered process which 
led to their actions to their human user, identify 
to the extent possible sources of uncertainty, 
and state any assumptions relied upon. 


A/IS should be programmed so that, under 
certain high risk situations where human 
decision-making is involved, they proactively 
inform users of uncertainty even when not 
asked. 


With any significant potential risk of 
economic or physical harm, designers should 
conspicuously and adequately warn users 

of the risk and provide a greater scope of 
proactive disclosure to the user. Designers 
should remain mindful that some risks 
cannot be adequately warned against and 
should be avoided entirely. 


To reduce the risk of A/IS that are 
unreasonably dangerous or that violate the 
law from being marketed and produced, 

we recommend lawmakers provide 
whistleblower incentives and protections. As 
in many industries, insiders may often be the 
first to know that the A/IS are acting illegally 
or dangerously. A well-crafted law to protect 
whistleblowers and allow a public interest 
cause of action would improve accountability 
and aid in prevention of intentional, reckless, 
or negligent misuses of A/IS. 


Government and industry groups should 
consider establishing standards that require 
A/IS to create logs (or other means of 
verification of their decision-making process) 
regarding key aspects of their operations 
and store those logs for a specified period 
of time. Designers should leverage current 
computer science regarding accountability 
and verifiability for code. New verification 
techniques may need to be developed 
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to overcome the technical challenges in 
verifiability and auditability of A/IS operations; 
A/IS oversight systems (“A/IS guardians”) or 


methods such as Quantitative Input Influence 





("QII") measures could facilitate this process. 
Making sure, ex ante, that such information 
is, or can be made, available will also provide 
a higher degree of trust in verifiability and a 
sense of transparency in A/IS operations. 


In Europe, the discussion on the so called 
"right to explanation” when automated 
decision-making occurs is important to 
address. Even though it is not yet guaranteed 
in Europe, future jurisprudence or Member 
State laws could grant individuals the right 
to ask for an explanation when a solely 
automated decision (e.g., refusal of an 
online credit application or e-recruiting 
practices) is being made about them that 
has legal or other significant effects. Such 

a right could provide a mechanism to 
increase the transparency and accountability 
of A/IS, and should therefore be seriously 
considered. In addition, other accountability 
enhancing tools such as ethical audits or 
certification schemes for algorithms should 
be explored. In addition, users should have 
the right to be informed, possibly through an 
interactive training program, on the areas of 
uncertainty, risks, and circumstances where 
safety or harm issues could arise, without 
this increasing user's accountability for A/IS 
decision-making consequences. 





Lawmakers on national and international 
levels should be encouraged to consider and 
carefully review a potential need to introduce 


10. 


new regulation where appropriate, including 
rules subjecting the market launch of new 
A/IS driven technology to prior testing and 
approval by appropriate national and/or 
international agencies. Companies should 
establish an A/IS ethics statement that 
includes statements about discrimination, 
addressing in that matter data-driven profiling 
and commitment to take measures to avoid 
user discrimination. In addition, companies 
should have internal systems that allow 
employees to identify and escalate issues 
related to discrimination in data and A/S. 
Laws should create whistleblower protection 
for those who can and wish to reveal explicit 
violation of discrimination law. In particular, 
a well-crafted law to protect whistleblowers 
and to allow a public interest cause of action 
would improve accountability and aid in 
prevention of intentional misuse of A/IS. 


The general public should be informed 
when articles/press releases related to 
political figures or issues are posted by an 
A/IS, such as a bot. 
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Affect is a core aspect of intelligence. Drives and emotions, such as excitement and 
depression, are used to coordinate action throughout intelligent life, even in species that 
lack a nervous system. We are coming to realize that emotions are not an impediment 

to rationality, arguably they are integral to rationality in humans. Emotions are one evolved 
mechanism for satisficing — for getting what needs to be done in the time available with 
the information at hand. Emotions are core to how individuals and societies coordinate 
their actions. Humans are therefore susceptible to emotional influence both positively 
and negatively. 


We would like to ensure that Al will be used to help humanity to the greatest extent 
possible in all contexts. In particular, artifacts used in society could cause harm either by 
amplifying or damping human emotional experience. It is quite possible we have reached 
a point where Al is affecting humans psychologically more than we realize. Further, even 
the rudimentary versions of synthetic emotions already in use have significant impact 

on how Al is perceived by policy makers and the general public. 


This subcommittee addresses issues related to emotions and emotion-like control in both 
humans and artifacts. Our working groups have put forward candidate recommendations 
on a variety of concerns: considering how affect varies across human cultures, the particular 
problems of artifacts designed for intimate relations, considerations of how intelligent 
artifacts may be used for “nudging,” how systems can support (or at least not interfere with) 
human flourishing, and appropriate policy concerning artifacts designed with their own 
affective systems. 
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* System Manipulation/Nudging/Deception 
* Systems Supporting Human Potential (Flourishing) 


* Systems with Their Own Emotions 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Systems Across Cultures 





Issue: 


Should affective systems 
interact using the norms 
appropriate for verbal and 
nonverbal communication 
consistent with the societal 
norms where they are located? 


Background 


Societies and therefore individuals around 

the world have different ways to maintain eye 
contact, express intentions through gestures, 
interpret silence, etc. These particularities could 
be incorporated into the affective systems in 
order to transmit the intended message. It would 
seem that an extensive study surrounding the 
norms/values of the community where the 
affective system will be deployed is essential 

to the system acceptability. 


Candidate Recommendations 


Any successful affective system should have 
a minimum set of ethical values/norms 

in its knowledge base that should be used 

in a specific cultural context. Some examples 
are listed below: 


creative 
commons 


Affective systems should be careful in using 
small talk. Although small talk is useful 

for acting friendly, some communities 

see people that use small talk as insincere 
and hypocritical, while other cultures see 
the opposite and tend to consider people 
that do not use small talk as unfriendly, 
uncooperative, rude, arrogant, or ignorant. 
Additionally speaking with proper vocabulary, 
grammar, and sentence structure is often 

in contrast to the typical interactions that 
people have. In many mature economies, 
the latest trend, TV show, or other media 
can significantly influence what is viewed as 
appropriate vocabulary and interaction style. 


Affective systems should recognize that 

the amount of personal space (proxemics) 
given is very important for human interaction. 
People from different cultures have different 
comfort zone distances to establish smooth 
communication. Crossing these limits without 
permission can transmit negative messages, 
such as hostile or sexual overtures. 


Eye contact is an essential component in 
social interaction for certain cultures, while 
for others, it is not essential and may even 
generate misunderstandings or conflicts. 

It is important to recognize this in the 
development of such systems. 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 164 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Affective Computing 





4. Hand gestures and other non-verbal 
interaction are very important for social 
interaction, but should be used with caution 
across cultures and should be acknowledged 
in the design of affective systems. For instance, 
although a “thumbs-up” sign is commonly 
used to indicate approval, in some countries 
this gesture can be considered an insult. 


5. Facial expressions are often used to 
detect emotions and facilitate emotional 
conversations. While it is tempting to 
develop A/IS that can recognize, analyze, 
and even display facial expressions for social 
interaction, it should be noted that facial 
expressions may not be universal across 
cultures and that an Al system trained with 
a dataset from one culture may not be 
readily usable in another culture. 


Further Resources 


The following documents/organizations can 
be used as additional resources to support 
the development of ethical affective systems. 


e Cotton, G. “Gestures to Avoid in Cross- 
Cultural Business: In Other Words, ‘Keep 
Your Fingers to Yourself!” Huffington Post, 
June 13, 2013. 











e “Cultural Intelligence & Paralanguage: 
Using Your Voice Differently Across Cultures. 
Sydney, Aus.: Culture Plus Consulting, 2016. 





n" 





* Cotton, G. Say Anything to Anyone, 
Anywhere: 5 Keys To Successful Cross- 
Cultural Communication. Hoboken, NJ: 
Wiley, 2013. 
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e Elmer, D. Cross-Cultural Connections: 
Stepping Out and Fitting In Around the 
World. Westmont, IL: InterVarsity Press, 
2002. 








e Price, M. “Facial Expressions—Including 
Fear—May Not Be as Universal as We 
Thought.” Science, October 17, 2016. 











Issue: 


Long-term interaction with 
affective artifacts lacking 
cultural sensitivity could 
alter the way people interact 
in society. 


Background 


Systems that do not have cultural knowledge 
incorporated into their knowledge base may 
change the way people interact, which may 
impact not only individuals, but also an entire 
society. Humans often use mirroring in order 
to understand and develop their principles and 
norms for behavior. At the same time, certain 
machine learning approaches focus on how 

to more appropriately interact with humans 

by mirroring human behavior. So learning via 
mirroring can go in both directions. If affective 
artifacts without cultural sensitivity interact with 
impressionable humans, they could alter the 
norms, principles, and therefore actions of that 
person. This creates a situation where the impact 
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of interacting with machines could significantly 
alter societal and cultural norms. For instance, 
children interacting with these systems can learn 
social and cultural values, which may be different 
from those present in their local community. 


Candidate Recommendations 


1. It is necessary to survey and analyze the 
long-term interaction of people with affective 
systems with different protocols and metrics 
to measure the modifications of habits, 
norms, and principles as well as the cultural 
and societal impacts. 


2. Responsible parties (e.g., parents, 
nurse practitioners, social workers, and 
governments) should be trained to detect 
the influence due to Al and in effective 
mitigation techniques. In the most extreme 
case it should always be possible to shut 
down harmful A/IS. 


Further Resources 


The following documents can be used as guides 
to support the development of ethical affective 
systems. 


e — Nishida, T., and C. Faucher. Modelling 
Machine Emotions for Realizing Intelligence: 
Foundations and Applications. Berlin, 
Germany: Springer-Verlag, 2010. 








e Pauleen, D. J. et al. “Cultural Bias in 
Information Systems Research and Practice: 
Are You Coming From the Same Place 
| Am?" Communications of the Association 
for Information Systems 17 (2006): 1-36. 
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e Bielby, J. “Comparative Philosophies in 


Intercultural Information Ethics.” Confluence: 
Online Journal of World Philosophies 2, 
no. 1 (2015): 233-253. 





* Bryson, J., “Why Robot Nannies Probably 


Won't Do Much Psychological Damage.” 

A commentary on an article by N. Sharkey 
and A. Sharkey, The Crying Shame of Robot 
Child Care Companions. Interaction Studies 
11, no. 2 (2010): 161-190. 








Sharkey, A., and N. Sharkey. “Children, 
the Elderly, and Interactive Robots.” IEEE 
Robotics & Automation Magazine 18, 
no. 1 (2011): 32-38. 





Issue: 


When affective systems 

are inserted across cultures, 
they could affect negatively 
the cultural/socio/religious 
values of the community 
where they are inserted. 


Background 


Some philosophers believe there are no universal 
ethical principles; instead they argue that ethical 
norms vary from society to society. Regardless 

of whether universalism or some form of ethical 
relativism is true, affective systems need to 
respect the values of the cultures within which 
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they are embedded. To some it may be that we 
should be designing affective systems which 

can reflect the values of those with which the 
systems are interacting. There is a high likelihood 
that when spanning different groups, the values 
imbued by the developer will be different from 
the operator or customer of that affective system. 
Differences between affective systems and 
societal values can generate conflict situations 
(e.g., gestures being misunderstood, or prolonged 
or inadequate eye contact) that may produce 
undesirable results, perhaps even physical violence. 
Thus, affective systems should adapt to reflect 
the values of the community (and individuals) 
where they will operate in order to avoid conflict. 


Candidate Recommendation 


Assuming the affective systems have a minimum 
subset of configurable ethical values incorporated 
in their knowledge base: 


1. They should have capabilities to identify 
differences between their values and the 
values of those they are interacting with and 
alter their interactions accordingly. As societal 
values change over time, any affective system 
needs to have the capability to detect this 
evolution and adapt its current ethical values 
to be in accordance with other people's values. 


2. Those actions undertaken by an affective 
system that are most likely to generate 


an emotional response should be designed 
to be easily changed. Similar to how 
software today externalizes the language 
and vocabulary to be easily changed based 
on location, affective systems should 
externalize some of the core aspects of 
their actions. 


Further Resources 


The following documents/organizations can 
be used as guides to support the development 
of ethical affective systems. 


e Bielby, J. “Comparative Philosophies in 
Intercultural Information Ethics.” Confluence: 
Online Journal of World Philosophies 2, 
no. 1 (2015): 233-253. 


e Velasquez, M., C. Andre, T. Shanks, and 
M. J. Meyer. “Ethical Relativism.”Markkula 
Center for Applied Ethics, Santa Clara, CA: 
Santa Clara University, August 1, 1992. 





e Culture reflects the moral values and 
ethical norms governing how people should 
behave and interact with others. “Ethics, 
an Overview.” Boundless Management. 





e Donaldson, T. “Values in Tension: Ethics 
Away from Home." Harvard Business Review. 
September—October 1996. 








e The Center for Nonviolent Communication. 
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When Systems Become Intimate 





Issue: 


Are moral and ethical 
boundaries crossed when the 
design of affective systems 
allows them to develop intimate 
relationships with their users? 


Background 


While robots capable of participating in an 
intimate relationship are not currently available, 
the idea that they could become intimate sexual 
partners with humans (e.g., sex robots) is one 
that captures the attention of the public and 
the media. Because the technology is already 
drawing much ethical scrutiny and may raise 
significant ethical concerns, it is important that 
policy makers and the professional community 
participate in developing guidelines for ethical 
research in this area. Part of the goal is to 
highlight potential ethical benefits and risks 

that may emerge, if and when affective systems 
develop intimacy with their users. Robots for 
use in the sex industry may help lessen human 
trafficking and the spread of STIs, but there 

is also the possibility that these systems could 
negatively impact human-to-human intimate 
relations. Human-to-human relations are currently 
viewed as being more rewarding, but also much 
more difficult to maintain than, for example, 

use of future robotic sex workers. 
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Candidate Recommendation 


As this technology develops, it is important to 
monitor research in this realm and support those 
projects that enhance the user's development 

of intimate relationships in positive and therapeutic 
ways while critiquing those that contribute 

to problematic intimate relations, specifically: 


1. Intimate systems must not be designed 
or deployed in ways that contribute to 
sexism, negative body image stereotypes, 
gender or racial inequality. 


2. Intimate systems must avoid the sexual/ 
psychological manipulation of the users 
of these systems unless the user is made 
aware they are being manipulated in this 
way (opt-in). 


3. Intimate systems should not be designed 
in a way that contributes to user isolation 
from other human companions. 


4. Designers of affective robotics, especially 
intimate systems, must foresee and publicly 
acknowledge that these systems can interfere 
with the relationship dynamics between 
human partners, causing jealousy or feelings 
of disgust to emerge between human 
partners. 


5. Intimate systems must not foster deviant 
or criminal behavior. Sex robots should not 
be built in ways that lead to the normalization 
of taboo, unethical, or illegal sexual practices, 
such as pedophilia or rape. 
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6. Commercially marketed Al should not be 
considered to be a person in a legal sense, 
nor marketed as a person. Rather its artifactual 
(authored, designed, and built deliberately) 
nature should always be made as transparent 
as possible, at least at point of sale and 
in available documentation (as noted 
in the Systems Supporting Human Potential 
Section below). 


Further Resources 


The following documents/organizations are 
provided for further research. 


e Levy, D. Love and Sex with Robots: The 
Evolution of Human-Robot Relationships. 
New York: HarperCollins Publishers, 2007 








* Scheutz, M. “The Inherent Dangers of 
Unidirectional Emotional Bonds Between 
Humans and Social Robots,” in Robot 
Ethics: The Ethical and Social Implications 
of Robotics, edited by P. Lin, K. Abney, 
and G. Bekey, 205. Cambridge, MA: MIT 
Press, 2011. 


e — Richardson, K. “The Asymmetrical 
‘Relationship’: Parallels Between Prostitution 
and the Development of Sex Robots.” 

ACM SIGCAS Newsletter, SIGCAS Computers 
& Society 45, no. 3 (2015): 290-293. 


* Sullins, J. P. “Robots, Love, and Sex: 
The Ethics of Building a Love Machine.” 
IEEE Transactions on Affective Computing 3, 
no. 4 (2012): 398-409. 


e Yeoman, l., and M. Mars. “Article Robots, 
Men and Sex Tourism.” Futures 44, no. 4 
(2012): 365-371. 
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e Campaign against Sex Robots. 


Whitby, B. “Do You Want a Robot Lover? 
The Ethics of Caring Technologies,” in Robot 
Ethics: The Ethical and Social Implications 
of Robotics, edited by P. Lin, K. Abney, 

and G. A. Bekey, 233-248. Cambridge, 

MA: MIT Press, 2012. 


e Danaher, J., and N. McArthur. Robot Sex: 
Sexual and Ethical Implications. Cambridge, 
MA: MIT Press, 2017. 





Issue: 


Can and should a ban 

or strict regulations be placed 
on the development of sex 
robots for private use or in 
the sex industry? 


Background 


The very idea of sex robots has sparked 
controversy even before many of these systems 
have become available. At this time, sex robots 
tend to be expensive love dolls made of silicone 
placed over a metal skeleton. These dolls 

can include robotic systems such as heating 
elements, sensors, movement, and rudimentary 
Al. The current state of the technology is a far cry 
from the sex robots imagined in novels and other 
media but they may just be the first step toward 
more advanced systems. There is ongoing debate 
around these systems. Critics are calling for strict 
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regulation or even a full ban on the development 
of this technology, while others argue that social 
value could be found by developing intimate 
robots, including on religious grounds. 


Sex robots are already used for prostitution 
and this trend is likely to continue in many 
regions of the world. Some researchers report 
that robot prostitutes will completely revolutionize 
the sex tourism industry by 2050. For example, 
by that time, Amsterdam's Red Light District 
may be dominated by a variety of android 
systems with various capabilities (Yeoman and 
Mars, 2012). However there are critics of the 
technology, including those who are calling 

for an outright ban. 


Despite being illegal, prostitution commonly 
occurs in many societies. Yet it is rarely done 
without creating deep ethical problems for 

the sex workers themselves and the societies 
in which they operate. Sex robots may alleviate 
some of these ethical concerns; for instance 

it has been argued that: 


1. Sex robots might be less likely to be a vector 
for the transmission of sexually transmitted 
infections (STIs). 


2. Sex robots could greatly lessen human 
trafficking of sex workers. 


3. Sex robots could be regulated by policies on 
controlling prices, hours of operations, sexual 
services, and other aspects of prostitution. 


However the technology can create serious 
ethical problems such as: 


1. This technology would likely further 
normalize the sex industry, and that typically 


means a further tendency to objectify women, 
given that the majority of clients for these 
technologies are heterosexual men. 


2. The availability of the technology could 
disrupt intimate relationships between 
human beings. 


Human sexuality is an important human 
activity, but it comes associated with difficult 
ethical issues related to power and desire. 
This means that robot sexual partners will 
always be an ethically contentious technology. 
A comprehensive/global ban on sex robots 
is unlikely given that a large market for these 
technologies may already exist and is part 

of the current demand for sex toys and 
devices. However, there are important issues/ 
considerations that the designers of these 
technologies need to consider. 


Candidate Recommendation 


1. We recommend regulation, not a ban, 
in accordance with cultural norms. 


2. Existing laws regarding personal imagery 
need to be reconsidered in light of 
robot sexuality. 


3. If it is proven through scientific studies that 
therapeutic uses of this technology could 
reduce recidivism in those who commit sex 
crimes, controlled use for those purposes 
only should be permitted, under legal and/ 
or medical supervision. 


4. Robot prostitution and sex tourism need 
to be monitored and controlled to fit local 
laws and policies. 
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Further Resources 


e Danaher, J., and N. McArthur. Robot Sex: 
Sexual and Ethical Implications. Cambridge, 
MA: MIT Press, 2017. 


e — Richardson, K. “The Asymmetrical 
‘Relationship’: Parallels Between Prostitution 
and the Development of Sex Robots.” 

ACM SIGCAS Newsletter, SIGCAS Computers 
& Society 45, no. 3 (2015): 290-293. 
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Sullins, J. P. “Robots, Love, and Sex: The Ethics 
of Building a Love Machine.” /EEE Transactions 
on Affective Computing 3, no. 4 (2012): 
398-409. 


Yeoman, I., and M. Mars. “Robots, Men 
and Sex Tourism.” Futures 44, no. 4 (2012): 
365-371. 


Campaign against Sex Robots. 
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System Manipulation/Nudging/Deception 





de Quintana Medina and Hermida Justo, 2016). 
Another key, and potentially more controversial, 


Issue: issue to be addressed is whether an affective 
Should affective systems system should be designed to nudge a user, 
be designed to nudge people and potentially intrude on individual liberty, 


for the user’s personal when doing so may benefit someone else. 


benefit and/or for the 


ee ern Candidate Recommendations 


1. Systematic analyzes are needed that examine 
the ethics of designing affective systems 
to nudge human beings prior to deployment. 


Background 

as an exercise of influence, with the intention recognize and distinguish between different 
to seize control and power at the person's types of nudges, including ones that seek 
expense. Thaler and Sunstein (2008) call the to promote beneficial social manipulation 
tactic of subtly modifying behavior a “nudge.” (e.g, healthy eating) versus others where 
Nudging mainly operates through the affective the aim is psychological manipulation or 
system. Making use of a nudge might be the exploitation of an imbalance of power 
considered appropriate in situations like teaching (e.g. for commercial purposes). 


children, treating drug dependency, healthcare, 
and when the global community benefits surpass 
individual benefits. Yet should affective systems 
be deployed to influence a user's behavior for 


3. Since nudging alters behavior implicitly, 
the resulting data on infantilization effects 
should be collected and analyzed. 


that person's own good? Nudging can certainly 4. Nudging in autonomous agents and robots 
trigger behaviors that worsen human health, must have an opt-in system policy with 
but could the tactic be used by affective explicit consent. 


systems to cue behaviors that improve it? Several 
applications are possible in health, well-being, 
education, etc. Yet a nudge could have opposite 
consequences on different people, with different 
backgrounds and preferences (White, 2013, 


5. Additional protections must be put in place 
for vulnerable populations, such as children, 
when informed consent cannot be obtained, 
or when it may not be a sufficient safeguard. 
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6. Nudging systems must be transparent and 
accountable, implying that data logging 
is required. This should include recording 
the user responses when feasible. 


Further Resources 


The following documents/organizations can 
be used as additional resources to support 
the development of ethical affective systems. 


e Thaler, R., and C. R. Sunstein. Nudge: 
Improving Decision about Health, Wealth 
and Happiness, New Haven, CT: Yale 
University Press, 2008. 


* Bovens, L. “The Ethics of Nudge,” in 
Preference change: Approaches from 
Philosophy, Economics and Psychology, 
edited by T. Griine-Yanoff and S. O. Hansson, 
207-219. Berlin, Germany: Springer. 


e de Quintana Medina, J., and P. Hermida Justo. 
“Not All Nudges Are Automatic: Freedom 
of Choice and Informative Nudges.” 
Working paper presented to the European 
Consortium for Political Research, Joint 
Session of Workshops, 2016 Behavioral 
Change and Public Policy, Pisa, Italy, 2016. 








e White, M. D. The Manipulation of Choice. 
Ethics and Libertarian Paternalism. 
New York: Palgrave Macmillan, 2013. 








* Scheutz, M. “The Affect Dilemma for Artificial 





Agents: Should We Develop Affective Artificial 





Agents?” /EEE Transactions on Affective 
Computing 3, no. 4 (2012): 424—433. 


* Grinbaum, A., R. Chatila, L. Devillers, 
J.-G. Ganascia, C. Tessier, and M. Dauchet. 
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"Ethics in Robotics Research: CERNA 
Recommendations,” IEEE Robotics and 
Automation Magazine 24, no. 3 (2017): 
139-145. 








e "Designing Moral Technologies: Theoretical, 
Practical, and Ethical Issues” Conference 
July 10-15, 2016, Monte Verita, Switzerland. 





Issue: 


Governmental entities often 
use nudging strategies, 

for example to promote the 
performance of charitable acts. 
But the practice of nudging for 
the benefit of society, including 
through the use of affective 
systems, raises a range 

of ethical concerns. 


Background 


A profoundly controversial practice that could 

be on the horizon is allowing a robot or another 
affective system to nudge a user for the good 

of society (Borenstein and Arkin, 2016). For 
instance, if it is possible that a well-designed 
robot could effectively encourage humans 

to perform charitable acts, would it be ethically 
appropriate for the robot to do so? This design 
possibility illustrates just one behavioral outcome 
that a robot could potentially elicit from a user. 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 173 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Affective Computing 





Given the persuasive power that an affective 
system may have over a user, ethical concerns 
related to nudging must be examined. This 
includes the significant potential for misuse. 


Candidate Recommendations 


As more and more computing devices subtly and 
overtly influence human behavior, it is important 
to draw attention to whether it is ethically 
appropriate to pursue this type of design pathway. 
There needs to be transparency regarding who 
the intended beneficiaries are, and whether 

any form of deception or manipulation is going 
to be used to accomplish the intended goal. 


Further Resources 


The following documents/organizations can 
be used as guides to support the development 
of ethical affective systems. 


* Borenstein, J., and R. Arkin. “Robotic Nudges: 
The Ethics of Engineering a More Socially 
Just Human Being.” Science and Engineering 
Ethics 22, no. 1 (2016): 31—46. 








e Borenstein, J., and R. C. Arkin. “Nudging 
for Good: Robots and the Ethical 
Appropriateness of Nurturing Empathy 
and Charitable Behavior.” A/ and Society 32, 
no. 4 (2016): 499-507. 
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Issue: 


A nudging system that does 
not fully understand the 
context in which it is operating 
may lead to unintended 
consequences. 


Background 


This kind of system needs to have sophisticated 
enough technical capabilities for recognizing 
the context in which it is applying nudging 
strategies. We could imagine a technical license 
(“permits”) (Omohundro, 2013). 


Candidate Recommendation 


1. When addressing whether affective systems 
should be permitted to nudge human 
beings, user autonomy is a key and essential 
consideration that must be taken into 
account. 


2. We recommend that when appropriate, 
an affective system that nudges human 
beings should have the ability to accurately 
distinguish between users, including 
detecting characteristics such as whether 
the user is an adult or a child. 


3. Affective systems with nudging strategies 
should be carefully evaluated, monitored, 
and controlled. 
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happiness. Given the diversity of views on the 
ethical appropriateness of deception, how should 


Further Resources 


The following documents/organizations can 
be used as guides to support the development 
of ethical affective systems. 


* Borenstein, J., and R. Arkin. “Robotic Nudges: 
The Ethics of Engineering a More Socially 
Just Human Being.” Science and Engineering 
Ethics 22, no. 1 (2016): 31-46. 








e Arkin, R. C., M. Fujita, T. Takagi, and R. 
Hasegawa. "An Ethological and Emotional 
Basis for Human-Robot Interaction.” Robotics 
and Autonomous Systems 42, no. 3—4 
(2003): 191-201. 








* Omohundro, S. “Autonomous Technology 
and the Greater Human Good." Journal 
of Experimental and Theoretical Artificial 
Intelligence 26, no. 3 (2014): 303-315. 











Issue: 


When, if ever, and under 
which circumstances 

is deception performed by 
affective systems acceptable? 


Background 


Deception is commonplace in everyday human- 
human interaction. According to Kantian ethics, 
it is never ethically appropriate to lie, while 
utilitarian frameworks would indicate that 

it can be acceptable when it increases overall 
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affective systems be designed to behave? 


Candidate Recommendations 


It is necessary to develop recommendations 
regarding the acceptability of deception in the 
design of affective autonomous agents with 
respect to when and under which circumstances, 
if any, it is appropriate. 


1. 


In general, deception is acceptable in an 
affective agent when it is used for the benefit 
of the person being deceived, not for the 
agent itself. For example, deception might 

be necessary in search and rescue operations, 
elder- or child-care. 


For deception to be used under any 
circumstance, a logical and reasonable 
justification must be provided by the designer, 
and this rationale must be approved by 

an external authority. 


Deception must follow an opt-in strategy 
and must be transparent to the user, i.e., the 
context under which the system is allowed 
to deceive. 


Further Resources 


Arkin, R. C., “Robots That Need to Mislead: 
Biologically-inspired Machine Deception.” 
IEEE Intelligent Systems 27, no. 6 (2012): 
60-75. 


Shim, J., and R. C. Arkin. “Other-Oriented 
Robot Deception: How Can a Robot's 
Deceptive Feedback Help Humans in HRI?” 
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Eighth International Conference on Social Safety, Security, and Rescue Robotics (SSRR 
Robotics (ICSR 2016), Kansas, Mo., 2015), West Lafayette, IN, October 2015. 
November 2016. 
¢ Shim, J., and R. C. Arkin. “A Taxonomy of 

¢ Shim, J., and R. C. Arkin. “The Benefits Robot Deception and its Benefits in HRI.” 
of Robot Deception in Search and Rescue: Proceedings of IEEE Systems, Man and 
Computational Approach for Deceptive Cybernetics Conference, Manchester England, 
Action Selection via Case-based Reasoning.” October 2013. 


2015 IEEE International Symposium on 
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Systems Supporting Human Potential 


(Flourishing) 





Issue: 1. 


Extensive use of artificial 
intelligence in society may make 
our organizations more brittle 
by reducing human autonomy 
within organizations, and 

by replacing creative, affective, 
empathetic components 

of management chains. 


Background 


As human workers are replaced by Al, their 
former employers (e.g., corporations and 
governments) may find they have eliminated 
the possibility of employees and customers 
discovering new equilibria outside the scope 

of what the organizations’ leadership originally 
foresaw. Even in ordinary, everyday work, a lack 
of empathy based on shared needs and abilities 
disadvantages not only the liberty of individuals 
but also the corporations and governments that 
exist to serve them, by eliminating opportunities 
for useful innovation. Collaboration requires 
sufficient commonality of collaborating 
intelligences to create empathy — the capacity 
to model the other's goals based on one’s own. 


Candidate Recommendations 


It is important that human workers within 

an organization have direct interactions 

with each other, rather than always being 
intermediated by affective systems (or other 
technology) which may filter out useful, 
unexpected communication. Similarly, 

we recommend human points of contact 

be available to customers and other 
organizations. 


In particular, although there will be many 
cases where Al is less expensive, more 
predictable, and easier to control than human 
employees, we recommend maintaining 

a core number of human employees at 
every level of decision-making with clear 
communication pathways. 


More generally, management and 
organizational theory should be extended 
to consider appropriate use of affective 
and autonomous systems to enhance their 
business model and the efficacy of their 
workforce. 


Further Resource 


The following document can be used as an 
additional resource to support the development 
of ethical affective systems. 
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+ Bryson, J. J. “Artificial Intelligence and 
Pro-Social Behavior,” in Collective Agency 
and Cooperation in Natural and Artificial 
Systems, edited by Catrine Misselhorn, 
281-306, Springer, 2015. 





Issue: 


The increased access to 

personal information about 

other members of our society, 
facilitated by artificial intelligence, 
may alter the human affective 
experience fundamentally, 
potentially leading to a 

severe and possibly rapid 

loss in individual autonomy. 


Background 


Theoretical biology tells us that we should expect 
increased communication — which Al facilitates 
— to increase group-level investment. This could 
have the effect of reducing individual autonomy 
and increasing in its place group-based identities. 
Candidate examples of this sort of social 
alteration include: 


1. Increased investment in monitoring and 
controlling children’s lives by parents. 


2. Decreased willingness to express opinions for 
fear of surveillance or long-term unexpected 
consequences. 
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3. Utilization of “customers” to perform 
basic corporate business processes such 
as data entry as a barter for lower prices, 
resulting also in reduced tax revenues. 


The loss of individual autonomy could lead 

to more fragmented or fragile societies, and 
(because diversity is associated with creativity) 
a reduction of innovation. This concern relates 
to issues of privacy and security, but also 

to social and legal liability for past expressions. 


Candidate Recommendations 


1. Organizations, including governments, must 
put a high value on individuals’ privacy and 
autonomy, including restricting the amount 
and age of data held on individuals. 


2. Educational countermeasures should be 
taken to encourage individuation and prevent 
loss of autonomy. 


Further Resources 


The following documents can be used as 
additional resources to support the development 
of ethical affective systems. 


e Bryson, J. J. “Artificial Intelligence and 
Pro-Social Behavior,” in Collective Agency 
and Cooperation in Natural and Artificial 
Systems, edited by Catrine Misselhorn, 
281-306, Springer, 2015. 


e Cooke, M.. “A Space of One's Own: 
Autonomy, Privacy, Liberty.” Philosophy & 
Social Criticism, 25, no. 1, (1999): 22-53. 
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* Roughgarden, J., M. Oishi, and E. Akçay. 
"Reproductive Social Behavior: Cooperative 
Games to Replace Sexual Selection.” 
Science 311, no. 5763 (2006): 965-969. 





Issue: 


A/IS may negatively affect 
human psychological and 
emotional well-being in 

ways not otherwise foreseen. 


Background 


A/IS has unprecedented access to human 
culture and human spaces — both physical and 


intellectual — for something that is not a human. 


A/IS may communicate via natural language, 
it may move in humanlike forms, and express 
humanlike identity. As such, it may affect 
human well-being in ways not yet anticipated. 
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Candidate Recommendations 


We recommend vigilance and research for 
identifying situations where A/IS are already 
affecting human well-being, both positively and 
negatively. We should look for evidence such as 
correlations between the increased use of A/IS 
and any suspected impacts. However, we should 
not be paranoid nor assume that correlation 
indicates causation. We recommend robust, 
ongoing, multidisciplinary research. 


Further Resource 


The following document can be used as an 
additional resource to support the development 
of ethical affective systems. 


* Kamewari, K., M. Kato, T. Kanda, H. Ishiguro, 
and K. Hiraki. “Six-and-a-Half-Month-Old 
Children Positively Attribute Goals to Human 
Action and to Humanoid-Robot Motion.” 
Cognitive Development 20, no. 2, (2005): 
303-320. 
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Systems With Their Own Emotions 





Issue: 


Synthetic emotions may 
increase accessibility of Al, 
but may deceive humans 
into false identification with 
Al, leading to overinvestment 
of time, money, trust, and 
human emotion. 


Background 


Deliberately constructed emotions are designed 
to create empathy between humans and artifacts, 
which may be useful or even essential for 
human-Al collaboration. However, this could 

lead humans to falsely identify with the A/IS, 

and therefore fail to realize that — unlike in 
evolved intelligence — synthetic emotions can 

be compartmentalized and even entirely removed. 
Potential consequences are over-bonding, 

guilt, and above all, misplaced trust. Because 
there is no coherent sense in which designed 
and engineered Al can be made to suffer, because 
any such affect, even if possible, could be avoided 
at the stage of engineering, or reengineered. 
Consequently, Al cannot be allocated moral 
agency or responsibility in the senses that have 
been developed for human sociality. 


creative 
commons 


Candidate Recommendations 


1. Commercially marketed Al should not 
be considered to be a person in a legal 
sense, nor marketed as a person. Rather 
its artifactual (authored, designed, and built 
deliberately) nature should always be made 
as transparent as possible, at least at point 
of sale and in available documentation. 


2. Some systems will, due to their application, 
require opaqueness in some contexts 
(e.g., emotional therapy). Transparency 
in such instances should not be necessarily 
during operation, but the systems’ working 
should still be available to inspection 
by responsible parties. 


Further Resources 


The following documents can be used as 
additional resources to support the development 
of ethical affective systems. 


e Arkin, R. C., P. Ulam, and A. R. Wagner. 
"Moral Decision-making in Autonomous 
Systems: Enforcement, Moral Emotions, 
Dignity, Trust and Deception," Proceedings 
of the IEEE 100, no. 3 (2012): 571-589. 


e Arkin, R., M. Fujita, T. Takagi, and R. Hasegawa. 
"An Ethological and Emotional Basis for 
Human-Robot Interaction,” Robotics and 
Autonomous Systems 42, no. 3—4 (2003): 
191-201. 
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e Arkin, R. C.. “Moving Up the Food Chain: 
Motivation and Emotion in Behavior-based 
Robots,” in Who Needs Emotions: The Brain 
Meets the Robot, edited by J. Fellous and 
M. Arbib. New York: Oxford University Press, 
2005. 


e Boden, M, J. Bryson, D. Caldwell, K. et al. 
“Principles of Robotics: Regulating Robots 
in the Real World.” Connection Science 29, 
no. 2 (2017): 124-129. 


e Bryson, J. J., M. E. Diamantis, and T. D. Grant. 


"Of, For, and By the People: The Legal Lacuna 


of Synthetic Persons.” Artificial Intelligence 
& Law 25, no. 3 (2017): 273-291. 
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Novikova, J., and L. Watts. “Towards Artificial 
Emotions to Assist Social Coordination in 
HRI.” International Journal of Social Robotics 
7 no. 1, (2015): 77-88. 


Scheutz, M. “The Affect Dilemma for Artificial 
Agents: Should We Develop Affective Artificial 
Agents?” /EEE Transactions on Affective 
Computing 3 (2012): 424—433. 


Sharkey, A., and N. Sharkey. “Children, the 
Elderly, and Interactive Robots.” /EEE Robotics 
& Automation Magazine 18.1 (2011): 32-38. 
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Autonomous and intelligent systems (A/IS) are a part of our society. The use of these 
new, powerful technologies promotes a range of social goods, and may spur development 
across the economies and society through its numerous applications, including in commerce, 
employment, healthcare, transportation, politics, privacy, public safety, national security, civil 
liberties, and human rights. To protect the public from adverse consequences, intended 

or otherwise, resulting from these applications, effective A/IS public policies and government 
regulations are needed. 


The goals of an effective A/IS policy center on the protection and promotion of safety, 
privacy, intellectual property rights, human rights, and cybersecurity, as well as the public 
understanding of the potential impact of A/IS on society. Without policies designed with 
these considerations in mind, there may be critical technology failures, loss of life, and 
high-profile social controversies. Such events could engender policies that unnecessarily 
stifle entire industries, or regulations that do not effectively advance public interest 

and protect human rights. 


To ensure that A/IS best serves the public interest, we believe that effective A/IS policies 
should embody a rights-based approach! that achieves five principal objectives: 


1. Support, promote, and enable internationally recognized legal norms 

2. Develop workforce expertise in A/IS technology 

3. Include ethics as a core competency in research and development leadership 
4. Regulate A/IS to ensure public safety and responsibility 


5. Educate the public on societal impacts of A/IS 


1 This approach is rooted in internationally recognized economic, social, cultural, and political rights. 


@greative 
commons 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 182 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 





As autonomous and intelligent systems (A/IS) become a greater part of our everyday lives, 
managing the associated risks and rewards will become increasingly important. Technology 
leaders and policy makers have much to contribute to the debate on how to build trust, 
prevent drastic failures, and integrate ethical and legal considerations into the design 

of A/IS technologies. 


Disclaimer: While we have provided recommendations in this document, it should be understood these are not formal policy 
recommendations endorsed by IEEE and do not represent a position or the views of IEEE but the informed opinions of Policy 
Committee members providing insights designed to provide expert directional guidance regarding A/IS. In no event shall IEEE 
or IEEE-SA Industry Connections Activity Members be liable for any errors or omissions, direct or otherwise, however caused, 
arising in any way out of the use of this work, regardless of whether such damage was foreseeable. 
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Objective: 


Ensure that A/IS support, 
promote, and enable 
internationally recognized 
legal norms. 


Background 


A/IS technologies have the potential to negatively 
impact internationally recognized economic, 
social, cultural, and political rights, through 
unintended outcomes or outright design 
decisions (as is the case with certain unmanned 
aircraft systems (Bowcott, 2013). In addition 

to the military application of A/IS, the domestic 
use of A/IS in predictive policing (Shapiro, 2017), 
banking (Garcia, 2017), judicial sentencing 
(Osoba and Welser, 2017), job hunting and hiring 
practices (Datta, Tschantz, and Datta, 2014), 
and even service delivery of goods (Ingold and 
Soper, 2016) can negatively impact human rights 
by automating certain forms of discrimination, 
inhibiting the right to assembly, freedom of 
expression, and access to information. To ensure 
A/IS are used as a force for good, it is crucial 

to formulate policies that prevent such violations 
of political, social, economic, and cultural rights. 


A/IS regulation, development, and deployment 
should, therefore, be based on international 
human rights standards and standards of 
international humanitarian laws (in the case 

of armed conflicts). This can be achieved if 
both states and private actors consider their 
responsibility to respectively protect and respect 
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internationally recognized political, social, 
economic, and cultural rights. For business actors, 
this means considering their obligation to respect 
international human rights, as laid out in the 

UN Guiding Principles for Business and Human 
Rights (OHCHR, 2011), also known as the 


Ruggie principles. 


When discussing the responsibility of private 
actors, the UN Guiding Principles on Business 
and Human Rights should be reflected. These 
principles have been widely referenced and 
endorsed by corporations and led to the adoption 
of several corporate social responsibility (CSR) 
policies in various companies. As such, 

they have led to a better understanding of the 
role of businesses in protection and promotion 

of human rights and ensured that the most 
crucial human values and legal standards of 
human rights are respected by A/IS technologists. 


Candidate Recommendations 


A rights-based approach means using the 
internationally recognized legal framework 

for human rights standards that is directed 

at accounting for the impact of technology 

on individuals. This framework also addresses 
inequalities, discriminatory practices, and the 
unjust distribution of resources. A/IS right-based 
policies will reflect the following principles: 


* Responsibility: The rights-based approach 
shall identify the right holders and the duty 
bearers, and ensure that duty bearers have 
an obligation to realize all human rights; 
this should guide the policy development 
and implementation of A/IS. 
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e Accountability: As duty bearers, states should 
be obliged to behave responsibly, seek to 
represent the greater public interest, and be 
open to public scrutiny of their A/IS policy. 


e Participation: the rights-based approach 
demands a high degree of participation 
of all interested parties. 


* Non-discrimination: Principles of non- 
discrimination, equality, and inclusiveness 
should underlie the practice of A/IS. 

The rights-based approach should also 
ensure that particular focus is given to 
vulnerable groups, to be determined locally, 
such as minorities, indigenous peoples, 

or persons with disabilities. 


e Empowerment: The rights-based approach 
to A/IS should empower right holders to 
claim and exercise their rights. 


* Corporate responsibility: Companies must 
ensure that when they are developing their 
technologies based on the values of a certain 
community, they do so only to the extent 
that such norms or values fully comply with 
the rights-based approach. Companies must 
also not willingly provide A/IS technologies 
to actors that will use them in ways that 
lead to human rights violations. 
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Further Resources 


Human rights-based approaches have 

been applied to development, education 
and reproductive health. See: the UN 
Practitioners’ Portal on Human Rights Based 


Programming. 





Bowcott, O. “Drone Strikes By Us May Violate 
International Law, Says UN.” The Guardian, 
October 18, 2013. 








Shapiro, A."Reform Predictive Policing.” 
Nature News 541, no. 7638 (2017): 458. 





Garcia, M. “How to Keep Your Al from Turning 
Into a Racist Monster.” Wired, April 21, 2017. 








Osoba, O. A., and W. Welser. “An Intelligence 
in Our Image: The Risks of Bias and Errors 
in Artificial Intelligence.” Santa Monica, CA: 
RAND Corporation, 2017. 








Datta, A., M. C. Tschantz, and A. Datta. 
"Automated Experiments on Ad Privacy 
Settings: A Tale of Opacity, Choice, and 
Discrimination.” arXiv:1408.6491 [Cs], 2014. 


Ingold, D., and S. Soper. “Amazon Doesn't 
Consider the Race of Its Customers. Should It?” 
Bloomberg, April 21, 2016. 





United Nations. Guiding Principles on 
Business and Human Rights: Implementing 
the United Nations “Protect, Respect and 
Remedy” Framework. United Nations Office 
of the High Commissioner of Human Rights. 
New York and Geneva: UN, 2011. 
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Candidate Recommendations 


A high level of technical expertise is required 
to create a public policy, legal, and regulatory 
environment that allows innovation to flourish 
while protecting the public and gaining public 
trust.' Policy makers and market leaders 
should pursue several strategies for developing 


Objective: 

Develop and make available 
to government, industry, 
and academia a workforce 


of well-qualified A/IS personnel. 


Background 


There is a clear consensus among private sector 
and academic stakeholders that effectively 
governing A/IS and related technologies requires 
a level of technical expertise that governments 
currently do not possess. Effective governance 
requires more experts who understand and 

can analyze the interactions between A/IS 
technologies, programmatic objectives, and 
overall societal values. With current levels of 
technical understanding and expertise, policies 
and regulations may fail to support innovation, 
adhere to national principles, and protect 

public safety. 


At the same time, the A/IS personnel should not 
only possess a necessary technology knowledge, 
but also receive adequate ethical training, and 
have access to other resources on human rights 
standards and obligations, along with guidance 
on how to make them a fundamental component 
of their work. 


this expertise: 


Expertise can be furthered by setting up 
technical fellowships, or rotation schemes, 
where technologists spend an extended time 
in political offices, or policy makers work with 
organizations that operate at the intersection 
of tech-policy, technical engineering, and 
advocacy (like the American Civil Liberties 
Union, Article 19, the Center for Democracy 
and Technology, or Privacy International). 
This will enhance the technical knowledge 
of policy makers and strengthen ties between 
political and technical communities, needed 
to make good A/S policy. 


A culture of sharing best practices around 
A/IS legislation, consumer protection, 
workforce transformation, and economic 
displacement stemming from A/IS-based 
automation should be fostered across 
borders. This can be done by doing 
exchange governmental delegation trips, 
transcontinental knowledge exchanges, 
and by building A/IS components into 
existing venues and efforts surrounding 
good regulation (General Data Protection 
Regulation (GDPR)). 


1 This recommendation concurs with the multiple recommendations of the United States National Science and Technology Council, 
One Hundred Year Study of Artificial Intelligence, Japan's Cabinet Office Council, European Parliament's Committee on Legal 
Affairs and others. 
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e In order to ensure that the next generation 
of policy makers is tech savvy, it is necessary 
to rely upon more than their “digital 
nativeness.” Because A/IS are evolving 
technologies, long-term educational 
strategies are needed, e.g., providing children 
access to coding and computer science 
courses starting from primary school, 
and extending into university or vocational 
courses. 


Further Resources 


e Holdren, J., and M. Smith. “Preparing for the 
Future of Artificial Intelligence.” Washington, 
DC: Executive Office of the President, 
National Science and Technology Council, 
2016. 





* Stanford University. “Artificial Intelligence 
and Life in 2030: One Hundred Year Study 
on Artificial Intelligence.” Stanford, CA: 
Stanford University, 2016. 











e “Japan Industrial Policy Spotlights Al, Foreign 
Labor.” Nikkei Asian Review, May 20, 2016. 








* Weng, Y.-H. “A European Perspective on 
Robot Law: Interview with Mady Delvaux- 
Stehres.” Robohub, July 15, 2016. 
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Objective: 


Support research and 
development needed to ensure 
continued leadership in A/IS. 


Background 


Greater national investment in ethical A/IS 
research and development would stimulate the 
economy, create high-value jobs, and improve 
governmental services to society. A/IS can 
significantly improve our societies: the use of 
A/IS in computer vision and human-computer 
interactions will have far-reaching implications. 
Intelligent robots will perform difficult and 
dangerous tasks that require human-like 
intelligence. Self-driving cars will revolutionize 
automobile transportation and logistics systems 
and reduce traffic fatalities. A/IS will improve 
quality of life through smart cities and decision 
support in healthcare, social services, criminal 
justice, and the environment. However, to ensure 
such a positive impact, more support for R&D, 
with a particular eye for the ethical impact 

of A/IS, is needed. 


Candidate Recommendations 


Investment in A/IS research and development 
(including ethical considerations) is essential 
to maximizing societal benefits, mitigating any 
associated risks, and enabling efficient and 
effective public sector investment. To enable 
efficient and effective public and private sector 
investment, there should be benchmarks 
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for A/IS technologies and applications with 
continuing focus on identifying promising future 
applications of A/IS. An important government 
role is to strategically educate the public and 
private sectors on key A/IS technologies and 
applications. We recommend the following: 


e Enable a cross-disciplinary research 
environment that encourages research 
on the fairness, security, transparency, 
understandability, privacy, and societal 
impacts of A/IS and that incorporates 
independent means to properly vet, audit, 
and assign accountability to the A/IS 
applications. 


e Governments should create research 
pools that incentivize research on A/IS that 
benefits the public, but which may not be 
commercially viable. 


Further Resources 
* — Kim, E. T. “How an Old Hacking Law Hampers 





the Fight Against Online Discrimination.” 
The New Yorker, October 1, 2016. 





e National Research Council. “Developments 
in Artificial Intelligence, Funding a Revolution: 
Government Support for Computing 
Research." Washington, DC: National 
Academy Press, 1999. 


e Chen, N., L. Christensen, K. Gallagher, 
R. Mate, and G. Rafert (Analysis Group). 
"Global Economic Impacts of Artificial 
Intelligence," February 25, 2016. 
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* The Networking and Information Technology 
Research and Development Program, 
“Supplement to the President's Budget, 
FY2017." NITRD National Coordination 
Office, April 2016. 








¢  Furber, S. B., F. Galluppi, S. Temple, and L. A. 
Plana. “The SpiNNaker Project.” Proceedings 
of the IEEE 102, no. 5 (2014): 652-665. 


e Markram, H. “The Human Brain Project.” 
Scientific American 306, no. 2 (June 2012): 
50-55. 


* L. Yuan. “China Gears Up in Artificial- 
Intelligence Race.” Wall Street Journal, 
August 24, 2016. 











Objective: 


Provide effective regulation 

of A/IS to ensure public 
safety and responsibility while 
fostering a robust Al industry. 


Background 


Governments must ensure consistent and 
appropriate policies and regulations for 

A/IS. Effective regulation should address 
transparency, understandability, predictability, 
and accountability of Al algorithms, risk 
management, data protection, and safety. 
Certification of systems involving A/IS is 
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a key technical, societal, and industrial issue. 
Good regulation encourages innovation, and 
harmonizing policy internationally will reduce 
barriers to trade. 


Good regulation can take many different forms, 
and appropriate regulatory responses are context- 
dependent. There is no one-size-fits-all for A/IS 
regulation, but it is important that such regulation 
is developed through an approach that is based 
on human rights? and has human well-being 

as a key goal. 


Candidate Recommendations 


e To ensure consistent and appropriate 
policies and regulations across governments, 
policymakers should seek informed input 
from a range of expert stakeholders, including 
academic, industry, and government 
officials, to consider questions related to the 
governance and safe employment of A/IS. 


e To foster a safe international community of 
A/IS users, policymakers should take similar 
work being carried out around the world into 
consideration. Due to the transnational nature 
of A/IS, globally synchronized policies can 
have a greater impact on public safety and 
technological innovation. 


e Law schools should offer interdisciplinary 
courses such as “Introduction to Al and 
Law” to reduce the gap between regulators, 
lawyers, and A/IS researchers and 
developers. 


Establish policies that foster the development 
of economies able to absorb A/IS, while 
providing broad job opportunities to those 
who might otherwise be alienated or 
unemployed. In addition, the continued 
development of A/IS talent should be 
fostered through international collaboration. 


Continue research into the viability of 
universal basic income. Such a non-conditional 
and government-provided addition to 
people's income might lighten the economic 
burden that comes from automation and 
economic displacement caused by A/IS. 


Ambiguity regarding whether and how 
proprietary A/IS may be reverse engineered 
and evaluated by academics, journalists, 
and other researchers can stifle innovation 
and public safety. Elimination of these 
impediments is essential. 


Further Resources 


Stanford University. “Artificial Intelligence 
and Life in 2030: One Hundred Year Study 
on Artificial Intelligence.” Stanford, CA: 
Stanford University, 2016. 











Calo, R. “The Case for a Federal Robotics 
Commission.” The Brookings Institution, 
2014. 





Mannes, A. “Institutional Options for Robot 
Governance,” 1—40, in We Robot 2016, 
Miami, FL, April 1-2, 2016. 





2 Human rights—based approaches have been applied to development, education, and reproductive health. 
See: the UN Practitioner's Portal on Human Rights Based Programming. 
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e Marchant, G. E., K. W. Abbott, and B. Allenby, 
Innovative Governance Models for Emerging 
Technologies. Cheltenham, U.K.: Edward 
Elgar Publishing, 2014. 


* Weng, Y. H. Y. Sugahara, K. Hashimoto, and 
A. Takanishi. “Intersection of ‘Tokku’ Special 
Zone, Robots, and the Law: A Case Study 
on Legal Impacts to Humanoid Robots.” 
International Journal of Social Robotics 7, 
no. 5 (2015): 841-857. 





Objective: 


Facilitate public understanding 
of the rewards and risks of A/IS. 


Background 


Perception drives public response. A/IS 
technologies and applications can both capture 
the imagination such as self-driving cars, and 
instill fear. Therefore, it is imperative for industry, 
academia, and government to communicate 
accurately both the positive potential of A/IS 
and the areas that require caution. Developing 
strategies for informing and engaging the public 
on A/IS benefits and challenges are critical to 
creating an environment conducive to effective 
decision-making. 


The success of A/IS technology depends on the 
ease with which people use and adapt to A/IS 
applications. While improving public understanding 
of A/IS technologies through education is 
becoming increasingly important, so is the need 
to educate the public about the social and 
cultural issues of A/IS. The way A/IS interact 
with final users, build cognitive models 

of their power and limits, and so help their 
adoption and sense of control, are key 
technological objectives. 


If society approaches these technologies primarily 
with fear and suspicion, societal resistance may 
result, impeding important work on ensuring 

the safety and reliability of A/IS technologies. 

On the other hand, if society is informed of 

the positive contributions and the opportunities 
A/IS create, then the technologies emerging from 
the field could profoundly transform 

society for the better in the coming decades.* 


Another major societal issue — and the subject 
of much ongoing debate — is whether A/IS 
should have, or could develop, any sense of 
ethical behavior. A/IS will require a commonly 
accepted sense of ethical behavior, or, at the 
very least, possess behaviors with ethical 
implications. Therefore, technology awareness 
and understanding of social and ethical issues 
of A/IS are new literacy skills society must 
embrace if A/IS applications are to be accepted 
and trusted as an integral part of modern living. 


3 One hundred year study of Al (Al100), Stanford University, August, 2016. 
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Candidate Recommendations 


+ Encourage A/IS development to serve the 
pressing needs of humanity by promoting 
dialogue and continued debate over the 
social and ethical implications of A/IS. 

To better understand the societal implications 
of A/IS, we recommend that funding be 
increased for interdisciplinary research on 
topics ranging from basic research into 
intelligence to principles on ethics, safety, 
privacy, fairness, liability, and trustworthiness 
of A/IS technology. Societal aspects should 
be addressed not only at an academic 

level but also through the engagement 

of business, public authorities, and policy 
makers. While technical innovation is a 

goal, it should not be prioritized over the 
protection of individuals. 


* Begin an international multi-stakeholder 
dialogue to determine the best practices 
for using and developing A/IS, and codify 
this dialogue into international norms and 
standards. Many industries, in particular 
system industries (automotive, air and 
space, defense, energy, medical systems, 
manufacturing) are going to be significantly 
changed by the surge of A/IS. A/IS algorithms 
and applications must be considered as 
products owned by companies, and therefore 
the companies must be responsible for the 
A/IS products not being a threat to humanity. 


e Empower and enable independent journalists 
and media outlets to report on A/IS, both 
by providing access to technical expertise 
and funding for independent journalism. 


creative 
commons 


Conduct media outreach to illustrate A/IS 
beneficial uses, and the important steps 
being taken to ensure safety and transparency. 
Public opinion related to trust, safety, privacy, 
employment, and the economy will drive 
public policy. It is critical to creating an 
environment conducive to effective decision- 
making, particularly as more government 
services come to rely on A/IS, that strategies 
are developed to inform and engage 

the public on Al benefits and challenges. 
Care must be taken to augment human 
interaction with A/IS and to avoid 
discrimination against segments of society. 


Further Resources 


Networking and Information Technology 
Research and Development (NITRD) 
Program. “The National Artificial Intelligence 
Research and Development Strategic Plan.” 
Washington, DC: Office of Science and 
Technology Policy, 2016. 








Saunders, J., P. Hunt, and J. S. Hollywood. 
“Predictions Put into Practice: A Quasi- 
Experimental Evaluation of Chicago's 
Predictive Policing Pilot,” Journal of 
Experimental Criminology 12, no. 347, 
(2016): 347-371. doi:10.1007/s11292- 
019272-0 











Edelman, B., and M. Luca. “Digital 
Discrimination: The Case of Airbnb.com.” 
Harvard Business School Working Paper 
14-054, 2014. 
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Garvie, C., A. Bedoya, and J. Frankle. 

"The Perpetual Line-Up: Unregulated Police 
Face Recognition in America.” Washington, 
DC: Georgetown Law, Center on Privacy 

& Technology, 2016. 








Chui M., and J. Manyika, “Automation, 
Jobs, and the Future of Work.” Seattle, WA: 
McKinsey Global Institute, 2014. 





The IEEE Global Initiative for Ethical 
Considerations in Artificial Intelligence 

and Autonomous Systems. Ethically Aligned 
Design: A Vision for Prioritizing Human 
Well-being with Artificial Intelligence and 
Autonomous Systems, Version 1. IEEE, 2016. 


Arkin, R. C. “Ethics and Autonomous 
Systems: Perils and Promises [Point of 
View].” Proceedings of the IEEE 104, 
no. 10, (1779-1781): 2016. 








Eurobarometer Survey on Autonomous 
Systems (published June 2015 by DG 
Connect) looks at Europeans’ attitudes to 
robots, driverless vehicles, and autonomous 
drones. The survey shows that those who 
have more experience with robots (at home, 
at work or elsewhere) are more positive 
toward their use. 
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The task of the Committee for Classical Ethics in Autonomous and Intelligent Systems 

is to apply classical ethics methodologies to considerations of algorithmic design in 
autonomous and intelligent systems (A/IS) where machine learning may or may not reflect 
ethical outcomes that mimic human decision-making. To meet this goal, the Committee 
has drawn from classical ethics theories as well as from the disciplines of machine ethics, 
information ethics, and technology ethics. 


As direct human control over tools becomes, on one hand, further removed, but on 

the other hand, more influential than ever through the precise and deliberate design 

of algorithms in self-sustained digital systems, creators of autonomous systems must 
ask themselves how cultural and ethical presumptions bias artificially intelligent creations, 
and how these created systems will respond based on such design. 


By drawing from over two thousand years’ worth of classical ethics traditions, the Classical 
Ethics in Autonomous and Intelligent Systems Committee will explore established ethics 
systems, addressing both scientific and religious approaches, including secular philosophical 
traditions such as utilitarianism, virtue ethics, and deontological ethics and religious- 
and-culture-based ethical systems arising from Buddhism, Confucianism, African Ubuntu 
traditions, and Japanese Shinto influences toward an address of human morality in the 
digital age. In doing so the Committee will critique assumptions around concepts such 

as good and evil, right and wrong, virtue and vice and attempt to carry these inquiries 

into artificial systems decision-making processes. 


Through reviewing the philosophical foundations that define autonomy and ontology, 

the Committee will address the potential for autonomous capacity of artificially intelligent 
systems, posing questions of morality in amoral systems, and asking whether decisions 
made by amoral systems can have moral consequences. Ultimately, it will address notions 
of responsibility and accountability for the decisions made by autonomous systems and 
other artificially intelligent technologies. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — Definitions for 
Classical Ethics in Autonomous 
and Intelligent Systems Research 





Issue: 


Assigning foundations 
for morality, autonomy, 
and intelligence. 


Background 


Classical theories of economy in the Western 
tradition, starting with Plato and Aristotle, 
embrace three domains: the individual, the 
family, and the polis. The forming of the individual 
character (ethos) is intrinsically related to others, 
as well as to the tasks of administration of work 
within the family (oikos) and eventually all this 
expands into the framework of the polis, or public 
space (poleis). This means that when we discuss 
ethical issues of autonomous and intelligent 
systems we should consider all three traditional 
economic dimensions that evolved in modernity 
into an individual morality disconnected from 
economics and politics. This disconnection was 
partly questioned by thinkers such as Adam 
Smith, Hegel, Marx, and others. In particular, 
Immanuel Kant's ethics located morality within 
the subject (see: categorical imperative) and 
separated morality from the outside world 
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and the consequences of being a part of the 
outside world. The moral autonomous subject 
of modernity became thus a worldless isolated 
subject. This process is important to understand 
in terms of ethics for artificial intelligence since 
it is, paradoxically, the kind of autonomy that 

is supposed to be achieved by intelligent 
machines in the very moment in which we, 
humans, begin to change our being into digitally 
networked beings. 


There lies a danger in uncritically attributing 
classical concepts of anthropomorphic autonomy 
to machines, including using the term artificial 
intelligence to describe them since, in the 
attempt to make them “moral” by programming 
moral rules into their behavior, we run the risk 

of assuming economic and political dimensions 
that do not exist, or that are not in line with 
contemporary human societies. As noted above, 
present human societies are being redefined 

in terms of digital citizenship via digital social 
networks. The present public debate about 

the replaceability of human work by intelligent 
machines is a symptom of this lack of awareness 
of the economic and political dimensions 

as defined by classical ethics, reducing ethical 
thinking to the “morality” of a worldless and 
isolated machine (a mimic of the modern subject). 
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Candidate Recommendations 


Via a return to classical ethics foundations, 
enlarge the discussion on ethics in 
autonomous and intelligent systems 
(A/IS) to include a critical assessment 

of anthropomorphic presumptions of ethics 
and moral rules for A/IS. Keep in mind 
that machines do not, in terms of classical 
autonomy, comprehend the moral or 

legal rules they follow, but rather move 
according to what they are programmed 
to do, following rules that are designed 

by humans to be moral. 


Enlarge the discussion on ethics for 

A/IS to include an exploration of the 
classical foundations of economy, outlined 
above, as potentially influencing current 
views and assumptions around machines 
achieving isolated autonomy. 


Further Resources 


Bielby, J., ed. “Digital Global Citizenship.” 
International Review of Information Ethics 
23 (November 2015). 





Bendel, O. “Towards a Machine Ethics.” 
Northwestern Switzerland: University of 
Applied Sciences and Arts, 2013. 





Bendel, O. “Considerations about the 
Relationship Between Animal and Machine 
Ethics.” Al & Society 31, no. 1 (2016): 
103-108. 








Capurro, R., M. Eldred, and D. Nagel. 
Digital Whoness: Identity, Privacy and 








Freedom in the Cyberworld. Berlin: 
Walter de Gruyter, 2013. 


e Chalmers, D. “The Singularity: A Philosophical 
Analysis.” Journal of Consciousness Studies 
17, (2010): 7-65. 








Issue: 
Distinguishing between 
agents and patients. 


Background 


Of concern for understanding the relationship 
between human beings and A/S is the 
uncritically applied anthropomorphistic approach 
toward A/IS that many industry and policy makers 
are using today. This approach erroneously blurs 
the distinction between moral agents and moral 
patients (i.e., subjects), otherwise understood 
as a distinction between “natural” self-organizing 
systems and artificial, non-self-organizing 
devices. As noted above, A/IS devices cannot, 
by definition, become autonomous in the sense 
that humans or living beings are autonomous. 
With that said, autonomy in machines, when 
critically defined, designates how machines act 
and operate independently in certain contexts 
through a consideration of implemented order 
generated by laws and rules. In this sense, 

A/IS can, by definition, qualify as autonomous, 
especially in the case of genetic algorithms 

and evolutionary strategies. However, attempts 
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to implant true morality and emotions, and thus 
accountability (i.e., autonomy) into A/IS is both 
dangerous and misleading in that it encourages 
anthropomorphistic expectations of machines 

by human beings when designing and interacting 
with A/IS. 


Thus, an adequate assessment of expectations 
and language used to describe the human-A/IS 
relationship becomes critical in the early stages 
of its development, where unpacking subtleties 
is necessary. Definitions of autonomy need 

to be clearly drawn, both in terms of A/IS and 
human autonomy. On one hand A/IS may in 
some cases manifest seemingly ethical and moral 
decisions, resulting for all intents and purposes 
in efficient and agreeable moral outcomes. 
Many human traditions, on the other hand, 

can and have manifested as fundamentalism 
under the guise of morality. Such is the 

case with many religious moral foundations, 
where established cultural mores are neither 
questioned nor assessed. In such scenarios, 
one must consider whether there is any 
functional difference between the level of 
autonomy in A/IS and that of assumed agency 
(the ability to choose and act) in humans via 
the blind adherence to religious, traditional, 

or habitual mores. The relationship between 
assumed moral customs (mores), the ethical 
critique of those customs (i.e., ethics), and the 
law are important distinctions. 


The above misunderstanding in definitions of 
autonomy arise in part because of the tendency 
for humans to shape artificial creations in their 
own image, and our desire to lend our human 
experience to shaping a morphology of artificially 
intelligent systems. This is not to say that such 


terminology cannot be used metaphorically, but 
the difference must be maintained, especially 
as A/IS begins to resemble human beings 

more closely. Terms like “artificial intelligence" 

or “morality of machines” can be used as 
metaphors, and it does not necessarily lend to 
misunderstanding to do so. This is how language 
works and how humans try to understand their 
natural and artificial environment. 


However the critical difference between human 
autonomy and autonomous systems involves 
questions of free will, predetermination, and 
being (ontology). The questions of critical 
ontology currently being applied to machines 

are not new questions to ethical discourse and 
philosophy and have been thoroughly applied 

to the nature of human being as well. John Stuart 
Mill, for example, is a determinist and claims that 
human actions are predicated on predetermined 
laws. He does, however, argue for a reconciliation 
of human free will with determinism through 

a theory of compatibility. Millian ethics provides 

a detailed and informed foundation for defining 
autonomy that could serve to help combat 
general assumptions of anthropomorphism 

in A/IS and thereby address the uncertainty 
therein (Mill, 1999). 


Candidate Recommendation 


When addressing the nature of “autonomy” 

in autonomous systems, it is recommended 
that the discussion first consider free will, civil 
liberty, and society from a Millian perspective 
in order to better grasp definitions of autonomy 
and to combat general assumptions of 
anthropomorphism in A/IS. 
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Further Resources 


e Capurro, Rafael. “Toward a Comparative 
Theory of Agents.” A/ & Society 27, no. 4 
(2012): 479-488. 








* King, William Joseph, and Jun Ohya. 
"The representation of agents: 
Anthropomorphism, agency, and intelligence.” 
Conference Companion on Human Factors 
in Computing Systems. ACM, 1996. 








* Hofkirchner, W. “Does Computing Embrace 
Self-Organization?” in Information and 
Computation, Essays on Scientific and 
Philosophical Understanding of Foundations 
of Information and Computation, edited 
by G. Dodig-Crnkovic, M. Burgin, 185-202. 
London: World Scientific, 2011. 








e International Center for Information Ethics. 





e Mill, J. S. On Liberty. London: Longman, 
Roberts & Green, 1869. 


* Verbeek, P.-P. What Things Do: Philosophical 
Reflections on Technology, Agency, and 
Design. University Park, PA: Penn State 
Press, 2010. 
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Issue: 


There is a need for 
an accessible classical 
ethics vocabulary. 


Background 


Philosophers and ethicists are trained in 
vocabulary relating to philosophical concepts 
and terminology. There is an intrinsic value 
placed on these concepts when discussing 
ethics and Al, since the layered meaning behind 
the terminology used is foundational to these 
discussions, and is grounded in a subsequent 
entrenchment of values. Unfortunately, using 
philosophical terminology in cross-discipline 
instances, for example, in conversation with 
technologists and policymakers is often ineffective 
since not everyone has the education to be able 
to encompass the abstracted layers of meaning 
contained in philosophical terminology. 


However, not understanding a philosophical 
definition does not detract from the necessity 

of its utility. While ethical and philosophical 
theories should not be over-simplified for popular 
consumption, being able to adequately translate 
the essence of the rich history of ethics traditions 
will go a long way in supporting a constructive 
dialogue on ethics and A/IS. As access and 
accessibility concerns are also intricately linked 
with education in communities, as well as 
secondary and tertiary institutions, society needs 
to take a vested interest in creating awareness 
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for government officials, rural communities, and 
school teachers. Creating a more “user-friendly” 
vocabulary raises awareness on the necessity and 
application of classical ethics to digital societies. 


Candidate Recommendation 


Support and encourage the efforts of groups 
raising awareness for social and ethics 
committees whose roles are to support ethics 
dialogue within their organizations, seeking 
approaches that are both aspirational and values- 
based. A/IS technologists should engage in 
cross-discipline exchanges whereby philosophy 
scholars and ethicists attend and present at 
non-philosophical courses. This will both raise 
awareness and sensitize non-philosophical 
scholars and practitioners to the vocabulary. 


Further Resources 


e Capurro, R. “Towards an Ontological 
Foundation of Information Ethics.” 
Ethics and Information Technology 8, 
no. 4 (2006): 175-186. 








e Flinders, D. J. “In Search of Ethical Guidance: 
Constructing a Basis for Dialogue 1.” 
Qualitative Studies in Education 5, no. 2 
(1992): 101-115. 








* Saldanha, G. S. “The Demon in the Gap 
of Language: Capurro, Ethics and Language 
in Divided Germany.” Information Cultures 
in the Digital Age. Wiesbaden, Germany: 
Springer Fachmedien, 2016. 253-268. 
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Issue: 


Presenting ethics to 
the creators of autonomous 
and intelligent systems. 


Background 


The question arises as to whether or not classical 
ethics theories can be used to produce meta- 
level orientations to data collection and data 

use in decision-making. The key is to embed 
ethics into engineering in a way that does not 
make ethics a servant, but instead a partner 

in the process. In addition to an ethics-in-practice 
approach, providing students and engineers with 
the tools necessary to build a similar orientation 
into their devices further entrenches ethical 
design practices. In the abstract this is not so 
difficult to describe, but very difficult to encode 
into systems. 


This problem can be addressed by providing 
students with job-aids such as checklists, 
flowcharts, and matrices that help them select 
and use a principal ethical framework, and then 
exercise use of those devices with steadily more 
complex examples. In such an iterative process, 
students will start to determine for themselves 
what examples do not allow for perfectly clear 
decisions, and in fact require some interaction 
between frameworks. Produced outcomes such 
as videos, essays, and other formats — such 

as project-based learning activities — allow 

for a didactical strategy which proves effective 

in artificial intelligence ethics education. 
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The goal is to provide students a means to 
use ethics in a manner analogous to how they 
are being taught to use engineering principles 
and tools. In other words, the goal is to help 
engineers tell the story of what they're doing. 


* Ethicists should use information flows and 
consider at a meta-level what information 


flows do and what they are supposed to do. 


* Engineers should then build a narrative 
that outlines the iterative process of ethical 
considerations in their design. Intentions 
are part of the narrative and provide 
a base to reflect back on those intentions. 


* The process then allows engineers to 
better understand their assumptions and 
adjust their intentions and design processes 
accordingly. They can only get to these 
by asking targeted questions. 


This process, one with which engineers are 
quite familiar, is basically Kantian and Millian 
ethics in play. 


The aim is to produce what in computer 
programming lexicon is referred to as a macro. 
A macro is code that takes other code as its 
input(s) and produces unique outputs. This 
macro is built using the Western ethics tradition 
of virtue ethics. 


Candidate Recommendation 


Find ways to present ethics where the 
methodologies used are familiar to engineering 
students. As engineering is taught as a collection 
of techno-science, logic, and mathematics, 
embedding ethical sensitivity into these objective 
and non-objective processes is essential. 


Further Resources 


+ Bynum, T. W., and S. Rogerson. Computer 
Ethics and Professional Responsibility. 
Malden, MA: Wiley-Blackwell, 2003. 


e Seebauer, E. G., and R. L. Barry. 
Fundamentals of Ethics for Scientists and 
Engineers. New York: Oxford University 
Press, 2001. 





* Whitbeck, C. “Teaching Ethics to Scientists 
and Engineers: Moral Agents and Moral 
Problems.” Science and Engineering Ethics 
1, no. 3 (1995): 299-308. 








* Zevenbergen, B. et al. “Philosophy Meets 
Internet Engineering: Ethics in Networked 
Systems Research.” GTC workshop outcomes 
paper. Oxford, U.K.: Oxford Internet Institute, 
University of Oxford, 2015. 











e Perez Á., and M. Angel, “Teaching Information 





Ethics.” International Review of Information 
Ethics 14 (12/2010): 23-28. 








e Verbeek, P-P. Moralizing Technology: 
Understanding and Designing the Morality 





of Things. Chicago: University of Chicago 
Press, 2011. 
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Issue: 


Access to classical ethics by 
corporations and companies. 


Background 


Many companies, from start-ups to tech giants, 
understand that ethical considerations in tech 
design are increasingly important, but are 

not quite sure how to incorporate ethics into 
their tech design agenda. How can ethical 
considerations in tech design become an 
integrated part of the agenda of companies, 
public projects, and research consortia? Many 
corporate workshops and exercises that attempt 
to consider ethics in technology practices present 
the conversation as a carte blanche for people 
to speak about their opinions, but serious 

ethical discussions are often lacking. As it stands, 
classical ethics is not accessible enough to 
corporate endeavors in ethics, and as such, are 
not applicable to tech projects. There is often, 
but not always, a big discrepancy between the 
output of engineers, lawyers, and philosophers 
when dealing with computer science issues 

and a large difference in how various disciplines 
approach these issues. While this is not true 

in all cases, and there are now several 
interdisciplinary approaches in robotics and 
machine ethics as well as a growing number 

of scientists that hold double and interdisciplinary 
degrees, there remains a vacuum for the wider 
understanding of classical ethics theories in the 
interdisciplinary setting. 
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Candidate Recommendation 


Bridge the language gap between technologists, 
philosophers, and policymakers. Understanding 
the nuances in philosophical language is 

critical to digital society from loT, privacy, and 
cybersecurity to issues of Internet governance. 


Further Resources 


* Bhimani, A. “Making Corporate Governance 
Count: The Fusion of Ethics and Economic 
Rationality.” Journal of Management & 
Governance 12, no. 2 (2008): 135-147. 








e Carroll, A. B. “A History of Corporate Social 
Responsibility.” in The Oxford Handbook of 
Corporate Social Responsibility, edited by 
Chrisanthi A., R. Mansell, D. Quah, and R. 
Silverstone. Oxford, U.K.: Oxford University 
Press, 2008. 








* Lazonick, W. “Globalization of the ICT 
Labor Force.” in The Oxford Handbook 
of Information and Communication 
Technologies, edited by Chrisanthi A., 
R. Mansell, D. Quah, and R. Silverstone. 
Oxford, U.K.: Oxford University Press, 2006. 








* IEEE P7000™, Model Process for Addressing 





Ethical Concerns During System Design. 
This standard will provide engineers and 





technologists with an implementable process 


aligning innovation management processes, 
IS system design approaches and software 
engineering methods to minimize ethical 
risk for their organizations, stakeholders and 
end users. The Working Group is currently 
in process, and is free and open to join. 
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Issue: 


Impact of automated systems 
on the workplace. 


Background 


The impact of A/IS on the workplace and the 
changing power relationships between workers 
and employers requires ethical guidance. 

Issues of data protection and privacy via big 

data in combination with the use of autonomous 
systems by employers is an increasing issue, 
where decisions made via aggregate algorithms 
directly impact employment prospects. The 
uncritical use of A/IS in the workplace in employee/ 
employer relations is of utmost concern due 

to the high chance for error and biased outcome. 


The concept of responsible research and 
innovation (RRI), a growing area, particularly 
within the EU, offers potential solutions to 
workplace bias and is being adopted by several 
research funders such as the EPSRC, who include 
RRI core principles in their mission statement. 
RRI is an umbrella concept that draws on classical 
ethics theory to provide tools to address ethical 
concerns from the outset of a project (design 
stage and onwards). 





Quoting Von Schomberg, “Responsible Research 
and Innovation is a transparent, interactive 
process by which societal actors and innovators 


become mutually responsive to each other with 
a view to the (ethical) acceptability, sustainability 
and societal desirability of the innovation process 
and its marketable products (in order to allow a 
proper embedding of scientific and technological 
advances in our society).”" 


When RRI methodologies are used in the ethical 
considerations of A/IS design, especially in 
response to the potential bias of A/IS in the 
workplace, theoretical deficiencies are then often 
exposed that would not otherwise have been 
exposed, allowing room for improvement in 
design at the development stage rather than from 
a retroactive perspective. RRI in design increases 
the chances of both relevance and strength 

in ethically aligned design. 


Candidate Recommendation 


It is recommended that through the application 
of RRI, as founded in classical ethics theory, 
research in A/IS design utilize available tools 
and approaches to better understand the design 
process, addressing ethical concerns from the 
very beginning of the design stage of the project, 
thus maintaining a stronger more efficient 
methodological accountability throughout. 


Further Resources 


e Burget, M., E. Bardone, and M. Pedaste. 
"Definitions and Conceptual Dimensions 
of Responsible Research and Innovation: 
A Literature Review.” Science and 
Engineering Ethics 23, no. 1 (2016): 1-9. 


1 Von Schomberg (2011) ‘Prospects for Technology Assessment in a framework of responsible research and innovation’ in: 
M. Dusseldorp and R. Beecroft (eds). Technikfolgen abschätzen lehren: Bildungspotenziale transdisziplinarer Methoden, 
Wiesbaden: Vs Verlag, in print, P.9. 
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Von Schomberg, R. “Prospects for Technology 


Assessment in a Framework of Responsible 
Research and Innovation,” in Technikfolgen 
Abschdtzen Lehren: Bildungspotenziale 
Transdisziplindrer Methode, 39-61, 
Wiesbaden, Germany: Springer VS, 2011. 


Stahl, B. C. et al. “From Computer Ethics 
to Responsible Research and Innovation in 
ICT: The Transition of Reference Discourses 











Informing Ethics-Related Research in 
Information Systems.” Information & 
Management 51, no. 6 (2014): 810-818. 








Stahl, B. C., and B. Niehaves. “Responsible 
Research and Innovation (RRI).” 





IEEE P7005™, Standard for Transparent 
Employer Data Governance is designed 

to provide organizations with a set of clear 
guidelines and certifications guaranteeing 
they are storing, protecting, and utilizing 
employee data in an ethical and transparent 
way. The Working Group is currently 

in process, and is free and open to join. 
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Section 2 — Classical Ethics From 
Globally Diverse Traditions 





Issue: 


The monopoly on ethics 
by Western ethical traditions. 


Background 


As human creators, our most fundamental 
values are imposed on the systems we design. 
It becomes incumbent on a global-wide 
community to recognize which sets of values 
guide the design, and whether or not A/IS 

will generate problematic (e.g., discriminatory) 
consequences without consideration of non- 
Western values. There is an urgent need to 
broaden traditional ethics in its contemporary 
form of “responsible innovation” (RI) beyond 
the scope of “Western” ethical foundations, 
e.g., utilitarianism, deontology, and virtue 
ethics; and include other traditions of ethics 

in RI, including those inherent to, for example, 
Buddhism, Confucianism, and Ubuntu traditions. 


However, this venture poses problematic 
assumptions even before the issue above can 
be explored, when, in classifying Western values, 
we also group together thousands of years 

of independent and disparate ideas originating 
from the Greco-Roman philosophical tradition 
with its Christian-infused cultural heritage. 


What is it that one refers to by the term Western 
ethics? By Western ethics, does one refer 

to philosophical ethics (ethics as a scientific 
discipline) or is the reference to Western morality? 


The West (however it may be defined) is an 
individualistic society, arguably more so than 
much of the rest of the world, and thus in some 
aspects should be even less collectively defined 
than say, “Eastern” ethical traditions. If one is 
referring to Western values, one must designate 
which values, and values of which persons 

and institutions. Additionally, there is a danger 
in intercultural information ethics (however 
unconsciously or instinctively propagated) to not 
only group together all Western traditions under 
a single banner, but to negatively designate any 
and all Western influence in global exchange 

to representing an abusive collective of colonial- 
influenced ideals. Just because there exists 

a monopoly of influence by one system over 
another does not mean that said monopoly is 
devoid of value, even for systems outside itself. 
In the same way that culturally diverse traditions 
have much to offer Western tradition(s), 

so too do they have much to gain from them. 





In order to establish mutually beneficial 
connections in addressing globally diverse 
traditions, it is of critical import to first properly 
distinguish between subtleties in Western 
ethics (as a discipline) and morality (as its 
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object or subject matter). It is also important 

to differentiate between philosophical ethics 
(as scientific ethics) and theological ethics. 

As noted above, the relationship between 
assumed moral customs (mores), the ethical 
critique of those customs (i.e., ethics), and the 
law is an established methodology in scientific 
communities. Western and Eastern philosophy 
are very different, as well as are Western and 
Eastern ethics. Western philosophical ethics uses 
scientific methods, e.g., the logical, discursive, 
dialectical approach (models of normative ethics) 
and the analytical and hermeneutical approach. 
The Western tradition is not about education 
and teaching of social and moral values, but 
rather about the application of fundamentals, 
frameworks, and explanations. However, several 
contemporary globally relevant community 
mores are based in traditional and theological 
moral systems, requiring a conversation around 
how best to collaborate in the design and 
programming of ethics in A/IS amidst differing 
ethical traditions. 


While experts in Intercultural Information Ethics, 
such as Pak-Hang Wong, highlight the dangers 
of the dominance of “Western” ethics in 

Al design, noting specifically the appropriation 
of ethics by liberal democratic values to the 
exclusion of other value systems, it should be 
noted that those same liberal democratic values 
are put in place and specifically designed to 
accommodate such differences. However, while 
the accommodation of differences are, in theory, 
accounted for in dominant liberal value systems, 
the reality of the situation reveals a monopoly 
of, and a bias toward, established Western 
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ethical value systems, especially when it comes 
to standardization. As Wong notes: 


Standardization is an inherently value-laden 
project, as it designates the normative criteria 
for inclusion to the global network. Here, 
one of the major adverse implications of the 
introduction of value-laden standard(s) of 
responsible innovation (RI) appears to be the 
delegitimization of the plausibility of RI based 
on local values, especially when those values 
come into conflict with the liberal democratic 
values, as the local values (or, the RI based 
on local values) do not enable scientists and 
technology developers to be recognized as 
members of the global network of research 
and innovation (Wong, 2016). 


It does however become necessary for those 
who do not work within the parameters of 
accepted values monopolies to find alternative 
methods of accommodating different value 
systems. Liberal values arose out of conflicts 
of cultural and subcultural difference and are 
designed to be accommodating enough to 
include a rather wide range of differences. 


Responsible innovation (RI) enables policy- 
makers, scientists, technology developers, and 
the public to better understand and respond 

to the social, ethical, and policy challenges 
raised by new and emerging technologies. 

Given the historical context from which RI 
emerges, it should not be surprising that the 
current discourse on RI is predominantly based 
on liberal democratic values. Yet, the bias toward 
liberal democratic values will inevitably limit 
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the discussion of RI, especially in the cases 
where liberal democratic values are not taken 
for granted. Against this background, it 

is important to recognize the problematic 
consequences of RI solely grounded on, 

or justified by, liberal democratic values. 


Candidate Recommendation 


In order to enable a cross-cultural dialogue 

of ethics in technology, discussions in ethics and 
A/IS must first return to normative foundations 
of RI to address the notion of “responsible 
innovation” from value systems not predominant 
in Western classical ethics, including nonliberal 
democratic perspectives. Pak-Hang Wong's paper, 
“Responsible Innovation for Decent Nonliberal 
Peoples: A Dilemma?” demonstrates the 
problematic consequences of RI solely grounded 
on, or justified by, liberal democratic values and 
should be consulted as a guide to normative 
foundations in RI. 


Further Resources 


e Bielby, J. “Comparative Philosophies in 
Intercultural Information Ethics.” Confluence: 
Journal of World Philosophies 2 (2016). 


* Hongladarom, S. “Intercultural Information 
Ethics: A Pragmatic Consideration.” 
Information Cultures in the Digital Age, 
191-206. Wiesbaden, Germany: Springer 
Fachmedien, 2016. 








* Rodriguez, L. G., and M. A. P. Alvarez. Ética 
Multicultural y Sociedad en Red. Fundación 
Telefonica, 2014. 
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* Wong, P.-H. "What Should We Share?: 
Understanding the Aim of Intercultural 
Information Ethics.” ACM SIGCAS Computers 
and Society 39, no. 3 (2009): 50-58. 











* Wong, P.-H. “Responsible Innovation for 
Decent Nonliberal Peoples: A Dilemma?” 
Journal of Responsible Innovation 3, no. 2 
(2016): 154-168. 








e Zeuschner, R. B. Classical Ethics, East and 
West: Ethics from a Comparative Perspective. 
Boston: McGraw-Hill, 2000. 








*  Mattingly-Jordan, S., Becoming a Leader 
in Global Ethics, IEEE, 2017. 








Issue: 


The application of classical 
Buddhist ethical traditions 
to Al design. 


Background 


According to Buddhism, ethics is concerned with 
behaving in such a way that the subject ultimately 
realizes the goal of Liberation. The question 
“How should | act?” is answered straightforwardly; 
one should act in such a way that one realizes 
Liberation (nirvana) in the future, achieving 

what in Buddhism is understood as “supreme 
happiness.” Thus Buddhist ethics are clearly 
goal-oriented. In the Buddhist tradition, people 
attain Liberation when they no longer endure 
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any unsatisfactory conditions, when they have 
attained the state where they are completely 
free from any passions, including desire, anger, 
and delusion (to name the traditional three), 
which ensnare one’s self against freedom. 

In order to attain Liberation, one engages oneself 
in mindful behavior (ethics), concentration 
(meditation), and what in Buddhism is deemed 
as wisdom, a term that remains ambiguous 

in Western scientific approaches to ethics. 


Thus ethics in Buddhism is concerned exclusively 
with how to attain the goal of Liberation, or 
freedom. In contrast to Western ethics, Buddhist 
ethics is not concerned with theoretical questions 
concerning the source of normativity or what 
constitutes the good life. What makes an action 

a “good” action in Buddhism is always concerned 
with whether the action leads, eventually, to 
Liberation or not. In Buddhism, there is no 
questioning as to why Liberation is a good thing. 
It is simply assumed. Such an assumption places 
Buddhism, and ethical reflection from a Buddhist 
perspective, in the camp of mores rather than 
scientifically led ethical discourse, and it is 
approached as an ideology or a worldview. 


While it is critically important to consider, 
understand, and apply accepted ideologies 
such as Buddhism in A/S, it is both necessary 
to differentiate the methodology from Western 
ethics, and respectful to Buddhist tradition 

not to require it be considered in a scientific 
context. Such assumptions put it at odds with, 
and in conflict with, the Western foundation of 
ethical reflection on mores. From a Buddhist 
perspective, one does not ask why supreme 
happiness is a good thing; one simply accepts 


it. The relevant question in Buddhism is not 
about methodological reflection, but about 
how to attain Liberation from the necessity 
for such reflection. 


Thus, Buddhist ethics contains potential for 
conflict with Western ethical value systems which 
are founded on ideas of questioning moral and 
epistemological assumptions. Buddhist ethics 

is different from, for example, utilitarianism, which 
operates via critical analysis toward providing the 
best possible situation to the largest number of 
people, especially as it pertains to the good life. 
These fundamental differences between the 
traditions need to be first and foremost mutually 
understood and then addressed in one form 

or another when designing A/IS that span 
cultural contexts. 


The main difference between Buddhist and 
Western ethics is that Buddhism is based 

upon a metaphysics of relation. Buddhist ethics 
emphasizes how action leads to achieving 

a goal, or in the case of Buddhism, the final 
Goal. In other words, an action is considered 

a good one when it contributes to realization of 
the Goal. It is relational when the value 

of an action is relative to whether or not it leads 
to the Goal, the Goal being the reduction and 
eventual cessation of suffering. In Buddhism, 
the self is constituted through the relationship 
between the synergy of bodily parts and mental 
activities. In Buddhist analysis, the self does 

not actually exist as a self-subsisting entity. 
Liberation, or nirvana, consists in realizing that 
what is known to be the self actually consists 

of nothing more than these connecting episodes 
and parts. To exemplify the above, one can draw 
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from the concept of privacy as oft explored via 
intercultural information ethics. The Buddhist 
perspective understands privacy as a protection, 
not of self-subsisting individuals, because such 
do not exist ultimately speaking, but a protection 
of certain values which are found to be necessary 
for a well-functioning society and one which 

can prosper in the globalized world. 


The secular formulation of the supreme 
happiness mentioned above is that of the 
reduction of the experience of suffering, or 
reduction of the metacognitive state of suffering 
as a result of lifelong discipline and meditation 
aimed at achieving proper relationships with 
others and with the world. This notion of the 
reduction of suffering is something that can 
resonate well with certain Western traditions, 
such as epicureanism and the notion of ataraxia, 
freedom from fear through reason and discipline, 
and versions of consequentialist ethics that 

are more focused on the reduction of harm. 

It also encompasses the concept of phronesis 
or practical wisdom from virtue ethics. 


Relational ethical boundaries promote ethical 
guidance that focuses on creativity and growth 
rather than solely on mitigation of consequence 
and avoidance of error. If the goal of the 
reduction of suffering can be formulated in 

a way that is not absolute, but collaboratively 
defined, this leaves room for many philosophies 
and related approaches to how this goal can be 
accomplished. Intentionally making space for 
ethical pluralism is one potential antidote 

to dominance of the conversation by liberal 
thought, with its legacy of Western colonialism. 


Candidate Recommendation 


In considering the nature of human and 
autonomous systems interactions, the above 
notion of “proper relationships” through 
Buddhist ethics can provide a useful platform 
that results in ethical statements formulated 

in a relational way, instead of an absolutist 

way, and is recommended as an additional 
methodology, along with Western values 
methodologies, to addressing human/computer 
interactions. 


Further Resources 


e Capurro, R. “Intercultural Information Ethics: 
Foundations and Applications.” Journal 
of Information, Communication & Ethics 
in Society 6, no. 2 (2008): 116. 








e Ess, C. “Ethical Pluralism and Global 
Information Ethics.” Ethics and Information 
Technology 8, no. 4 (2006): 215-226. 








* Hongladarom, S. “Intercultural Information 
Ethics: A Pragmatic Consideration,” in 
Information Cultures in the Digital Age 
edited by K. M. Bielby, 191-206. Wiesbaden, 
Germany: Springer Fachmedien Wiesbaden, 
2016. 








* Hongladarom, S. et al. “Intercultural 
Information Ethics.” International Review 
of Information Ethics 11 (2009): 2-5. 
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e Nakada, M. “Different Discussions on 
Roboethics and Information Ethics Based 
on Different Contexts (Ba). Discussions 
on Robots, Informatics and Life in the 
Information Era in Japanese Bulletin Board 
Forums and Mass Media.” Proceedings 
Cultural Attitudes Towards Communication 
and Technology (2010): 300-314. 




















e Mori, Ma. The Buddha in the Robot. 
Suginami-ku, Japan: Kosei Publishing, 1989. 








Issue: 


The application of 
Ubuntu ethical traditions 
to A/IS design. 


Background 


In his article, “African Ethics and Journalism 
Ethics: News and Opinion in Light of Ubuntu,” 
Thaddeus Metz frames the following question: 
“What does a sub-Saharan ethic focused on the 
good of community, interpreted philosophically 
as a moral theory, entail for the duties of various 
agents with respect to the news/opinion media”? 
(Metz, 2015, 1). When that question is applied 
to A/IS) viz: “If an ethic focused on the good of 
community, interpreted philosophically as a moral 
theory, is applied to autonomous and intelligent 
systems, what would the implications be on the 
duties of various agents"? Agents in this regard 
would therefore be the following: 
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1. Members of the A/IS research community 
2. A/IS programmers/computer scientists 

3. A/IS end-users 

4. Autonomous and intelligent systems 


Ubuntu is a Sub-Saharan philosophical tradition. 
Its basic tenet is that a person is a person 
through other persons. It develops further in the 
notions of caring and sharing as well as identity 
and belonging, whereby people experience their 
lives as bound up with their community. A person 
is defined in relation to the community since the 
sense of being is intricately linked with belonging. 
Therefore, community exists through shared 
experiences and values: “to be is to belong to 

a community and participate” also motho ke 
motho ka batho “A person is a person because 
of other people.” 


Very little research, if any at all, has been 
conducted in light of Ubuntu ethics and A/IS, 
but its focus will be within the following moral 
domains: 


1. Between the members of the A/IS research 
community 


2. Between the A/IS community/programmers/ 
computer scientists and the end-users 


3. Between the A/IS community/programmers/ 
computer scientists and A/IS 


4. Between the end-users and A/IS 


5. Between A/IS and A/IS 
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Considering a future where A/IS will become 
more entrenched in our everyday lives, one must 
keep in mind that an attitude of sharing one's 
experiences with others and caring for their well- 
being will be impacted. Also by trying to ensure 
solidarity within one’s community, one must 
identify factors and devices that will form part 

of their lifeworld. If so, will the presence of A/IS 
inhibit the process of partaking in a community, 
or does it create more opportunities for doing 
so? One cannot classify A/IS as only a negative 
or disruptive force; it is here to stay and its 
presence will only increase. Ubuntu ethics must 
come to grips with and contribute to the body 
of knowledge by establishing a platform for 
mutual discussion and understanding. 


Such analysis fleshes out the following suggestive 
comments of Desmond Tutu, renowned former 
chair of South Africa’s Truth and Reconciliation 
Commission, when he says of Africans, “(we say) 
a person is a person through other people... 

| am human because | belong” (Tutu, 1999). 

| participate, | share. Harmony, friendliness, 

and community are great goods. Social harmony 
is for us the summum bonum — the greatest 
good. Anything that subverts or undermines 


this sought-after good is to be avoided (2015:78). 


In considering the above, it is fair to state that 
community remains central to Ubuntu. In situating 
A/IS within this moral domain, it will have to 
adhere to the principles of community, identity 
and solidarity with others. While virtue ethics 
questions the goal or purpose of A/IS and 
deontological ethics questions the duties, the 
fundamental question asked by Ubuntu would 


be “how does A/IS affect the community in 
which it is situated”? This question links with 

the initial question concerning the duties of 

the various moral agents within the specific 
community. Motivation becomes very important, 
because if A/IS seek to detract from community 
it will be detrimental to the identity of this 
community, i.e., in terms of job losses, poverty, 
lack in education and skills training. However, 
should A/IS seek to supplement the community, 
i.e., ease of access, support systems, etc., then 
it cannot be argued that it will be detrimental. 

It therefore becomes imperative that whosoever 
designs the systems must work closely both with 
ethicists and the target community/audience/ 
end-user to ascertain whether their needs are 
identified and met. 


Candidate Recommendations 


e — It is recommended that a concerted effort 
be made toward the study and publication 
of literature addressing potential relationships 
between Ubuntu ethical traditions and 
A/IS value design. 


* A/IS designers and programmers must 
work closely with the end-users and target 
communities to ensure their design aims 
are aligned with the needs of the end-users 
and target communities. 


Further Resources 


* Lutz, D. W. “African Ubuntu Philosophy and 
Global Management.” Journal of Business 
Ethics 84 (2009): 313-328. 
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e Metz, T. “African Ethics and Journalism Ethics: 





News and Opinion in Light of Ubuntu,” 
Journal of Media Ethics: Exploring Questions 
of Media Morality 30 no. 2 (2015): 74-90. 
doi: 10.1080/23736992.2015.1020377 





* Tutu, D. No Future Without Forgiveness. 
London: Rider, 1999. 








Issue: 


The application of Shinto- 
influenced traditions 
to A/IS design. 


Background 


Alongside the burgeoning African Ubuntu 
reflections on A/IS, other indigenous techno- 
ethical reflections boast an extensive engagement. 
One such tradition is Japanese Shinto indigenous 
spirituality, (or, Kami-no-michi), often cited as the 
very reason for Japanese robot and autonomous 
systems culture, a culture more prevalent in 
Japan than anywhere else in the world. Popular 
Japanese Al, robot and video-gaming culture 

can be directly connected to indigenous Shinto 
tradition, from the existence of kami (spirits) 

to puppets and automata. 


The relationship between A/IS and a human 
being is a personal relationship in Japanese 
culture and, one could argue, a very natural 
one. The phenomenon of relationship in Japan 
between humans and automata stands out as 


unique to technological relationships in world 
cultures, since the Shinto tradition is arguable the 
only animistic and naturalistic tradition that can 
be directly connected to contemporary digital 
culture and A/IS. From the Shinto perspective, 
the existence of A/IS, whether manifested 
through robots or other technological 
autonomous systems, is as natural to the 
world as are rivers, forests, and thunderstorms. 
As noted by Spyros G. Tzafestas, author of 
Roboethics: A Navigating Overview, "Japan's 
harmonious feeling for intelligent machines 
and robots, particularly for humanoid ones,” 
(Tzafestas, 2015, 155) colors and influences 
technological development in Japan, especially 
robot culture. 


The word Shinto can be traced to two Japanese 
concepts, Shin, meaning spirit, and “to”, the 
philosophical path. Along with the modern 
concept of the android, which can be traced 
back to three sources — one, to its Greek 
etymology that combines “avdpac”: andras 
(man) and gynoids, “yuv”: gyni (woman); 
two, via automatons and toys as per U.S. 
patent developers in the 1800s, and three to 
Japan, where both historical and technological 
foundations for android development have 
dominated the market since the 1970s — 
Japanese Shinto-influenced technology culture 
is perhaps the most authentic representation 
of the human-automaton interface. 


Shinto tradition is an animistic religious 
tradition, positing that everything is created 
with, and maintains, its own spirit (kami) and 
is animated by that spirit, an idea that goes 

a long way to defining autonomy in robots from 
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a Japanese viewpoint. This includes on one hand, 
everything that Western culture might deem 
natural, including rivers, trees, and rocks, and 

on the other hand, everything artificially (read: 
artfully) created, including vehicles, homes, 

and automata (i.e., robots). Artifacts are as 

much a part of nature in Shinto as are animals, 
and are considered naturally beautiful rather 

than falsely artificial. 


A potential conflict between Western concepts 
of nature and artifact and Japanese concepts 
of the same arises when the two traditions 

are compared and contrasted, especially in 

the exploration of artificial intelligence. Where 
in Shinto, the artifact as artificial represents 
creation and authentic being (with implications 
for defining autonomy), the same is designated 
as secondary and oft times unnatural, false, 
and counterfeit in Western ethical philosophical 
tradition, dating back to Platonic and Christian 
ideas of separation of form and spirit. In both 
traditions, culturally presumed biases define our 
relationships with technology. While disparate 
in origin and foundation, both Western classical 
ethics traditions and Shinto ethical influences 
in modern A/IS have similar goals and outlooks 
for ethics in A/IS, goals that are centered 

in relationship. 


@greative 
commons 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 211 


Candidate Recommendation 


Where Japanese culture leads the way in 

the synthesis of traditional value systems and 
technology, we recommend that efforts in 

A/IS ethics explore the Shinto paradigm as 
representative, though not necessarily as directly 
applicable, to global efforts in understanding 
and applying traditional and classical ethics 
methodologies to ethics for A/IS. 


Further Resources 


* — Holland-Minkley, D. F. “God in the Machine: 
Perceptions and Portrayals of Mechanical 
Kami in Japanese Anime.” PhD Diss. 
University of Pittsburgh, 2010. 











e Jensen, C. B., and A. Blok. “Techno-Animism 
in Japan: Shinto Cosmograms, Actor-Network 
Theory, and the Enabling Powers of Non- 
Human Agencies.” Theory, Culture & Society 
30, no. 2 (2013): 84-115. 











* Tzafestas, S. G. Roboethics: A Navigating 
Overview. Cham, Switzerland: Springer, 2015. 
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Section 3 — Classical Ethics for 


a Technical World 





Issue: 
Maintaining human autonomy. 


Background 


Autonomous and intelligent systems present 

the possibility for a digitally networked intellectual 
capacity that imitates, matches, and supersedes 
human intellectual capacity, including, among 
other things, general skills, discovery, and 
computing function. In addition, A/IS can 
potentially acquire functionality in areas traditionally 
captured under the rubric of what we deem 
unique human and social ability. While the larger 
question of ethics and Al looks at the implications 
of the influence of autonomous systems in 

these areas, the pertinent issue is the possibility 
of autonomous systems imitating, influencing, 
and then determining the norms of human 
autonomy. This is done through the eventual 
negation of independent human thinking and 
decision-making, where algorithms begin to 
inform through targeted feedback loops what it 

is we are and what it is we should decide. Thus, 
how can the academic rigor of traditional ethics 
speak to the question of maintaining human 
autonomy in light of algorithmic decision-making? 


How will Al and autonomous systems influence 
human autonomy in ways that may or may not 
be advantageous to the good life, and perhaps 
even if advantageous, may be detrimental at the 
same time? How do these systems affect human 
autonomy and decision-making through the use 
of algorithms when said algorithms tend to inform 
(“in-form”) via targeted feedback loops? 


Consider, for example, Google's autocomplete 
tool, where algorithms attempt to determine 
one's search parameters via the user's initial 
keyword input, offering suggestions based on 
several criteria including search patterns. In this 
scenario, autocomplete suggestions influence, 
in real-time, the parameters the user phrases 
their search by, often reforming the user's 
perceived notions of what it was they were 
looking for in the first place, versus what they 
might have actually originally intended. 


Targeted algorithms also inform as per emerging 
loT applications that monitor the user's routines 
and habits in the analog world. Consider for 
example that our bio-information is, or soon will 
be, available for interpretation by autonomous 
systems. What happens when autonomous 
systems can inform the user in ways the user is 
not even aware of, using one's bio-information 
in targeted advertising campaigns that seek 

to influence the user in real-time feedback loops 
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based on the user's biological reactions (pupil 
dilation, body temperature, emotional reaction), 
whether positive or negative, to that very same 
advertising, using information about our being 
to in-form (and re-form) our being? 


On the other hand, it becomes important not 

to adopt dystopian assumptions concerning 
autonomous machines threatening human 
autonomy. The tendency to think only in negative 
terms presupposes a case for interactions 
between autonomous machines and human 
beings, a presumption not necessarily based 

in evidence. Ultimately the behavior of algorithms 
rests solely in their design, and that design 

rests solely in the hands of those who designed 
them. Perhaps more importantly, however, 

is the matter of choice in terms of how the user 
chooses to interact with the algorithm. Users 
often don’t know when an algorithm is interacting 
with them directly, or their data which acts as 

a proxy for their identity. The responsibility for 
the behavior of algorithms remains with both the 
designer and the user and a set of well-designed 
guidelines that guarantee the importance of 
human autonomy in any interaction. As machine 
functions become more autonomous and 

begin to operate in a wider range of situations, 
any notion of those machines working for 

or against human beings becomes contested. 
Does the machine work for someone in 
particular, or for particular groups but not for 
others, and who decides on the parameters? 

The machine itself? Such questions become key 
factors in conversations around ethical standards. 
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Candidate Recommendation 


An ethics by design methodology is the first 
step to addressing human autonomy in Al, 
where a critically applied ethical design of 
autonomous systems preemptively considers 
how and where autonomous systems may 
or may not dissolve human autonomy. 


The second step is a pointed and 

widely applied education curriculum 

that encompasses school age through 
university, one based on a classical ethics 
foundation that focuses on providing 
choice and accountability toward digital 
being as a priority in information and 
knowledge societies. 


Further Resources 


van den Berg, B. and J. de Mul. "Remote 
Control. Human Autonomy in the Age of 
Computer-Mediated Agency,” in: Autonomic 
Computing and Transformations of Human 
Agency. Philosophers of Law Meeting 
Philosophers of Technology, edited by 
Mireille Hildebrandt and Antoinette Rouvroy, 
46-63. London: Routledge, 2011. 

















Costa, L. “A World of Ambient Intelligence,” 
Chapter 1 in Virtuality and Capabilities 

in a World of Ambient Intelligence, 15—41. 
Cham, Switzerland: Springer International, 
2016. 
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* Verbeek, P.-P. “Subject to Technology 
on Autonomic Computing and Human 
Autonomy,” in The Philosophy of Law Meets 
the Philosophy of Technology: Autonomic 
Computing and Transformations of Human 
Agency, edited by. M. Hildebrandt and 
A. Rouvroy. New York: Routledge, 2011. 








Issue: 

Applying goal-directed behavior 
(virtue ethics) to autonomous 
and intelligent systems. 


Background 


Initial concerns regarding A/IS also include 
questions of function, purpose, identity, and 
agency, a continuum of goal-directed behavior, 


with function being the most primitive expression. 


How can classical ethics act as a regulating force 
in autonomous technologies as goal-directed 
behavior transitions from being externally set by 
operators to being indigenously set? The question 
is important not just for safety reasons, but for 
mutual productivity. If autonomous systems are 
to be our trusted, creative partners, then we 
need to be confident that we possess mutual 
anticipation of goal-directed action in a wide 
variety of circumstances. 


A virtue ethics approach has merits for 
accomplishing this even without having to posit 
a “character” in an autonomous technology, since 


it places emphasis on habitual, iterative action 
focused on achieving excellence in a chosen 
domain or in accord with a guiding purpose. 
At points on the goal-directed continuum 
associated with greater sophistication, virtue 
ethics become even more useful by providing 
a framework for prudent decision-making that 
is in keeping with the autonomous system's 
purpose, but allows for creativity in how to 
achieve the purpose in a way that still allows 
for a degree of predictability. An ethics that 
does not rely on a decision to refrain from 
transgressing, but instead to prudently pursue 
a sense of purpose informed by one’s identity, 
might provide a greater degree of insight into 
the behavior of the system. 


Candidate Recommendation 


Program autonomous systems to be able to 


recognize user behavior as being those of specific 


types of behavior and to hold expectations as 
an operator and co-collaborator whereby both 
user and system mutually recognize the decisions 


of the autonomous system as virtue ethics based. 


Further Resources 


e Lennox, J. G. “Aristotle on the Biological 


Roots of Virtue.” Biology and the Foundations 


of Ethics, edited by J. Maienschein and 
M. Ruse, 405—438. Cambridge, U.K.: 
Cambridge University Press, 1999. 


* Boden, M. A., ed. The Philosophy of Artificial 
Life. Oxford, U.K.: Oxford University Press, 
1996. 





@greative 
commons 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 214 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Classical Ethics in A/IS 


© 


creative 
commons 


e Coleman, K. G.. “Android Arete: Toward 
a Virtue Ethic for Computational Agents.” 
Ethics and Information Technology 3, no. 4 
(2001): 247-265. 











A requirement for 
rule-based ethics 
in practical programming. 


Background 


Research in machine ethics focuses on simple 
moral machines. It is deontological ethics 

and teleological ethics that are best suited 

to the kind of practical programming needed 
for such machines, as these ethical systems 
are abstractable enough to encompass ideas 
of non-human agency, whereas most modern 
ethics approaches are far too human-centered 
to properly accommodate the task. 





In the deontological model, duty is the point 
of departure. Duty can be translated into rules. 
It can be distinguished into rules and meta 
rules. For example, a rule might take the form 
“Don't lie!”, whereas a meta rule would take 
the form of Kant's categorical imperative: 

“Act only according to that maxim whereby 
you can, at the same time, will that it should 
become a universal law.” 


A machine can follow simple rules. Rule-based 
systems can be implemented as formal systems 
(also referred to as axiomatic systems), and 


in the case of machine ethics, a set of rules 

is used to determine which actions are morally 
allowable and which are not. Since it is not 
possible to cover every situation by a rule, an 
inference engine is used to deduce new rules 
from a small set of simple rules (called axioms) 
by combining them. The morality of a machine 
comprises the set of rules that are deducible 
from the axioms. 


Formal systems have an advantage since 
properties such as decidability and consistency 
of a system can be effectively examined. 

If a formal system is decidable, every rule 

is either morally allowable or not, and the 
“unknown" is eliminated. If the formal system 
is consistent, one can be sure that no two rules 
can be deduced that contradict each other. 

In other words, the machine never has moral 
doubt about an action and never encounters 
a deadlock. 


The disadvantage of using formal systems is 
that many of them work only in closed worlds 
like computer games. In this case, what is not 
known is assumed to be false. This is in drastic 
conflict with real world situations, where rules 
can conflict and it is impossible to take into 
account the totality of the environment. In other 
words, consistent and decidable formal systems 
that rely on a closed world assumption can 

be used to implement an ideal moral framework 
for a machine, yet they are not viable for real 
world tasks. 


One approach to avoiding a closed world 
scenario is to utilize self-learning algorithms, 
such as case-based reasoning approaches. 
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Here, the machine uses “experience” in the 
form of similar cases that it has encountered 
in the past or uses cases which are collected 
in databases. 


In the context of the teleological model, 

the consequences of an action are assessed. 
The machine must know the consequences of 
an action and what the action’s consequences 
mean for humans, for animals, for things in the 
environment, and, finally, for the machine itself. 
It also must be able to assess whether these 
consequences are good or bad, or if they are 
acceptable or not, and this assessment is not 
absolute: while a decision may be good for 
one person, it may be bad for another; while 

it may be good for a group of people or for 

all of humanity, it may be bad for a minority 

of people. An implementation approach 

that allows for the consideration of potentially 
contradictory subjective interests may be 
realized by decentralized reasoning approaches 
such as agent-based systems. In contrast to this, 
centralized approaches may be used to assess 


the overall consequences for all involved parties. 
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Candidate Recommendation 


By applying the classical methodologies of 
deontological and teleological ethics to machine 
learning, rules-based programming in A/IS 

can be supplemented with established praxis, 
providing both theory and a practicality toward 
consistent and decidable formal systems. 


Further Resources 


Bendel, O. Die Moral in der Maschine: 
Beiträge zu Roboter-und Maschinenethik. 
Heise Medien, 2016. 








Bendel, O. "LADYBIRD: the Animal-Friendly 
Robot Vacuum Cleaner.” The 2017 AAAI 
Spring Symposium Series. Palo Alto, CA: 
AAAI Press, 2017. 








Fisher, M., L. Dennis, and M. Webster. 
"Verifying Autonomous Systems.” 
Communications of the ACM 56, no. 9 
(2013): 84-93. 





McLaren, B. M. “Computational Models of 
Ethical Reasoning: Challenges, Initial Steps, 
and Future Directions.” IEEE Intelligent 
Systems 21, no. 4 (2006): 29-37. 











Perez Alvarez, M. A. "Tecnologías de la 
Mente y Exocerebro o las Mediaciones del 








Aprendizaje," 2015. 
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Mixed reality could alter our very notions of identity and reality over the next generation, as 
these technologies infiltrate more and more aspects of our lives, from work to education, 
from socializing to commerce. An autonomous and intelligent systems (A/IS) backbone 
that would enable real-time personalization of this illusory world raises a host of ethical and 
philosophical questions, especially as the technology moves from headsets to much more 
subtle and integrated sensory enhancements. This committee has been working to discover 
the methodologies that could provide this future with an ethical skeleton and the assurance 
that the rights of the individual, including control over one's increasingly multifaceted 
identity, will be reflected in the encoding of this evolving environment. While augmented, 
virtual, and mixed reality deal primarily with technological environments, A/IS technologies 
utilizing and influencing user data in these environments present unique ethical challenges 
society must face today to avoid negative unintended consequences that could harm 
innovation and greatly decrease human well-being tomorrow. 


Our Committee has created the following sections within mixed reality to help address 
these ethical challenges: 


1. Social Interactions 
2. Mental Health 


3. Education and Training 
4. The Arts 





5. Privacy Access and Control 


It is our hope that by addressing these challenges today, we can create a more positive, 
ethical, and intentional reality, whatever the environment. 


Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — Social Interactions 


The nature of mediated reality and the ability 

for individuals to alter their identity (or for their 
identity to be altered by other actors) means that 
social interactions will definitely be affected by 
the widespread adoption of mixed reality. 





Issue: 


Within the realm of A/IS- 
enhanced mixed reality, how 
can we evolve, harness, and 
not eradicate the positive 
effects of serendipity? 


Background 


In the real world, bumping into a stranger when 
your GPS breaks means you may meet your 

life partner. However, in the digital and virtual 
spheres, algorithms that have been programmed 
by design may eliminate genuine randomness 
from our human experience. What do we stand 
to lose when we code “frictions” or randomness 
out of our lives that may cause discomfort, but 
can also bring joy and growth? 


For several years now, we have witnessed how 
online systems automatically sculpt the reality we 
encounter. Two major forces have come together: 
the commercial imperative to give customers 
what they want, and the desire of customers to 
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use technology to make their lives easier, more 
comfortable, more controllable, safer, and less 
disruptive. These tendencies have always existed, 
but out of the last decade of digital media 

has emerged a rudimentary version of what 

the coming intelligent mixed-reality world will 
probably look like, in terms of the use of personal 
data and A/IS to create an environment in which 
the user has actually become the product. 


Eli Pariser's “filter bubble” is the inevitable result 
of consumers’ desire to get what they want 
enabled by an industry that naturally wants to 
create products that will sell. This effect, however, 
will become qualitatively different and much 
more profound when the curated content goes 
from a window on a laptop to becoming a full- 
time part of the physical world. 


Is an augmented or virtual world an improvement 
over the physical world when it can be controlled 
in ways possible only in an illusion? Or does 

it become a denatured place, a software 
concoction more inclined toward order and 
predictability than freedom and invention? 

What would widespread use of such technology 
have on individuals, society, and politics over 

the long term? 


In a physical city, a great deal of life, good and 
bad, is open to randomness, chance, risk, and 
the constant threat of encountering behavior one 
would rather not encounter. At the same time, 
there are unpredictable and often inspirational 
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experiences that could not happen elsewhere, 
and over time can broaden one's embrace of 
human diversity along all axes. In a gated suburb, 
by contrast, these qualities are markedly reduced. 
We trade inspiration for control. Qualities are 
traded off for other qualities. 


Creating the digital version of the gated community 
will happen naturally — they are both designed 
systems. But how can developers create MR/A/IS 
experiences that allow users what might be called 
the city option — the ability to live in, for example, e 
a virtual world that somehow mimics the truly 
unpredictable aspects many people love about 
cities? Can such a simulation have the same effect 
as the "real thing” if there's no actual risk of serious 
unpleasantness? Could the degree of “serendipity” 
be dialed in by the user? 


Candidate Recommendation 


1. Upon entering any virtual realm, individuals 
should be provided information about the 
nature of algorithmic tracking and mediation 
within any environment. This will allow 
not only for consent regarding the use of 
their personal data, but for improved trust 
between individuals and creators of these 
environments regarding user experience. This 
could also include a “serendipity on or off” 
button allowing a user to express their desire 
for randomness as well. 


2. Work with the MR/A/IS development 
community to address this challenge and try 
to make it a standard part of the conversation 
from the very beginning of MR/A/IS-related 
project development. 


Further Resources 


Kefalidou, G., and S. Sharples. “Encouraging 
Serendipity in Research: Designing | 
Technologies to Support Connection-Making,” 
International Journal of Human-Computer 
Studies 89 (2016): 1-23. 








Harford, T. Messy: The Power of Disorder to 
Transform Our Lives, New York: Riverhead 
Books, 2016. 


Pariser, E. The Filter Bubble: How the New 
Personalized Web Is Changing What We 
Read and How We Think. New York: Penguin 
Books, 2011. 











Rabin, S., J. Goldblatt, and F. Silva. “Advanced 
Randomness Techniques for Game Al: 
Gaussian Randomness, Filtered Randomness, 
and Perlin Noise” in Game Al Pro: Collected 
Wisdom of Game Al Professionals, edited 

by S. Rabin, 29—43. Natick, MA: Taylor & 
Francis, 2013. 
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Issue: 


What happens to cultural 
institutions in a mixed reality, 
Al-enabled world of illusion, 
where geography is largely 
eliminated, tribe-like entities 
and identities could spring up 
spontaneously, and the notion 
of identity morphs from physical 
certainty to virtuality? 


Background 


When an increasing amount of our lives is 

spent in a photorealistic and responsive world 

of software, what will happen to actual human 
contact, which might always remain undigitizable 
in meaningful ways? When an illusory world 

is vastly more pleasant and fulfilling than the 
physical alternative, will there be a significant 
population who choose to live exclusively, or 
who spend at least a majority of their time, in 

a synthetic world of their own making? Opting 

in and out will be central to the coming digital 
experiences; but what happens with the opposite 
— when people choose to opt-out of the “real” 
world in favor of illusion? 


MR/A/IS technology could be especially 
meaningful in allowing people to create a physical 
appearance that more closely reflects who they 
are. For example, it could help transgender 
persons reconcile their physical appearance with 
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their identity. Is the optimal digital representation 
of a person the externally observable physical 
facade, or an illusion better aligned to the 
individual's self-image and identity? 


While the benefits of spending time in alternate 
realities could include increasing empathy toward 
others or discovering aspects of your individuality 
that could positively affect your identity (in 
either real or virtual reality), there are multiple 
benefits of human interaction, both physical and 
emotional, that could be affected adversely if too 
much time is spent within realities of one’s own 
creation. 


Candidate Recommendation 


Provide widespread educational classes on the 
benefit of positive human connection/touch. 
This could involve fields including emotional 
intelligence or positive psychology. 


Further Resources 


e Fredrickson, B. L. “Your Phone Versus Your 
Heart” (Sunday Review). New York Times, 
March 23, 2013. 








* McGonigal, J., and J. Whelan. Reality Is 
Broken: Why Games Make Us Better and 
How They Can Change the World. New York: 
Penguin Books, 2011. 








* Turkle, S. Alone Together: Why We Expect 
More from Technology and Less from Each 
Other. New York: Basic Books, 2011. 








e Pasqualini, l., J. Llobera, and O. Blanke. 
“Seeing’ and ‘Feeling’ Architecture: How 
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Bodily Self-Consciousness Alters Architectonic 
Experience and Affects the Perception of 
Interiors.” Frontiers in Psychology 4, (2013): 
354. 








e Hershfield, H., D. W. Goldstein, W. Sharpe, 
J. Fox, L. Yeykelis, L. Carstensen et al. 
"Increasing Saving Behavior Through Age- 
Progressed Renderings of the Future Self.” 
Journal of Marketing Research 48, no. SPL, 
(2011): S23-S37. 











Issue: 


With alternative realities at 
reach, we will have alternative 
ways of behaving individually 
and collectively, and perceiving 
ourselves and the world around 
us. These new orientations 
regarding reality could enhance 
an already observed tendency 
toward social reclusiveness 
that detaches many from our 
common reality. Could such 

a situation lead to an 
individual opting out of 
“societal engagements?” 
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Background 


The availability of VR and AR could lead to 
permanent disengagement from society that can 
have far-reaching implications on fertility rates, 
the economy, and alter existing social fabrics. 
People may choose to disengage. 


With mixed reality, our notions of time will be 
multi-modal and as such will have a societal 
impact in terms of culture, relationships, and 
perception of the self. We might be able to 
manipulate our perceptions of time and space so 
as to experience, or re-experience, interactions 
that would otherwise be impossible. With 
alternative realities in reach, people may inhabit 
them to avoid facing problems they encounter in 
real life. 


Candidate Recommendation 


Research and potentially consider the 
reconstruction of our social contract as alternative 
mixed societies, including the concept of present 
virtual and physical beings that will potentially 
emerge from alternative realities. 


Further Resources 


e Petkova, V., and Ehrsson, H. (2008). “If 
| Were You: Perceptual Illusion of Body 
Swapping.” PLoS ONE 3, no. 12 (2008): 1-9. 





e Rainie, L., and J. Anderson. “The Evolution 
of Augmented Reality and Virtual Reality.” 
December 14, 2008. 





e Peck, T., S. Seinfeld, S. Aglioti, and M. Slater. 
“Putting Yourself in the Skin of a Black Avatar 
Reduces Implicit Racial Bias.” Consciousness 
and Cognition 22, no. 3 (2013): 779-787. 
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Issue: 


The way we experience (and 
define) physical reality on a daily 
basis will soon change. 


Background 


VR and AR technologies are very popular in China, 
for example, where dedicated experimental 
zones are gaining significant traction. VR cafes 
are changing the way we interact with people 
around us and offer experiences that rival 

movie theaters, theme parks, and travel. For 
example, VR applications have been introduced 
to attractions’ sites and are used to provide an 
interactive experience for tourists who can better 
acquaint themselves with new environments 

and attractions. This also changes the way we 
experience our physical reality on a daily basis. In 
addition, augmented-reality enhancement over 
the next generation will become ubiquitous in 
the physical environment, from our homes to city 
streets, and will inevitably alter our view of what 
constitutes reality or physical certainty. 


Candidate Recommendation 


Create widespread education about how the 
nature of mixed reality will affect our social 
interactions to avoid widespread negative 
societal consequences. 
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Further Resources 


e  Madary, M., and T. K. Metzinger. “Real_ 
Virtuality: A Code of Ethical Conduct. 
Recommendations for Good Scientific 
Practice and the Consumers of VR- 
Technology.” Frontiers in Robotics and Al 3 
(February 19, 2016). 














Issue: 


We may never have to say 
goodbye to those who have 
graduated to a newer 
dimension (i.e., death). 


Background 


Whether we will have the ability to keep our 
consciousness alive via software or create an 
avatar copy of ourselves or loved ones, there is 
the very real possibility we will see a person's 
representation after death as we know it. While 
the decision to upload one's consciousness 

or represent oneself as an avatar after death 

is a deeply personal one, there are multiple 

legal, societal, and cultural issues to deal with 
(e.g., identity, next of kin) to avoid confusion or 
potential manipulation of “living” family members 
and friends. In the future, if one’s consciousness 
is still “alive” in some sense and able to engage in 
human activities, is that person still legally alive? 
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Candidate Recommendation 


New forms of societal norms around traditional 
death will need to be created for governments 
(updating forms of identity such as passports, 
etc.) along with cultural mores (sending family 
and friends cards letting them know a certain 
person's consciousness has transferred from 
carbon-based to silicon). 


Further Resource 


e Rothblatt, M. Virtually Human: The Promise— 
and the Peril—of Digital Immortality, New 
York: St, Martin’s Press, 2014. 











Issue: 


Mixed reality changes the way we 
interact with society and can also 
lead to complete disengagement. 


Background 


The increasing popularity of VR and AR dedicated 
zones and their use in public sites in China, for 
example, is changing the way individuals interact 
with each other. Where friends and colleagues 
would previously emphasize eye contact and 
physical proximity as a way of establishing trust 
and a sense of cohesion, MR will change the way 
we perceive the people we interact with. They 
may be judged based on their avatars, their ability 
to navigate this new reality, and their willingness 
to interact via MR. The inability or choice whether 
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to use MR might exclude an individual from a 
working environment or from a new connected 
socializing platform. 


MR can also be used to disengage from 

one’s environment. Individuals can choose to 

go back in time and relive happy memories 
recorded by MR technology (whether real or 
not), go on vacation to a venue miles and years 
away, or immerse themselves in some virtual 
entertainment — all without leaving their chair 
and without interacting with other people. This 
can lead to the disengagement of individuals 
even when in the company of others, as virtual 
interactions can supplement and surpass human 
interaction in the user experience they offer. 

In this way, individuals can “fulfill” their social 
needs without reciprocating those of others. 
This artificial “fulfillment” of basic social needs 
through fully immersive technologies might have 
unpredicted implications on the very fabric of 
society, especially by changing the way humans 
interact with each other. 


Candidate Recommendations 


MR content providers should be well aware 

of the ramifications of offering alternative 
social interactions that do not require a human 
counterpart, or severely limit key social cues. 


Further Resource 


* Kim, M. “The Good and the Bad of Escaping- 
to Virtual Reality.” The Atlantic, February 18, 
2015. 
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Issue: 


A/IS, artificial consciousness, 
and augmented/mixed reality 
has the potential to create a 
parallel set of social norms. 


Background 


Mixed reality poses the potential to redefine and 
reset many human social norms. Traditionally 
human norms have been established by 
influences such as religion, politics, and 
economics, to name a few. The interactions 
between people and augmented/mixed reality 
could generate an entirely different set of 
norms created entirely by the designer of the 
mixed reality. There is likely to be opportunity to 
positively influence and enhance new norms via 
augmented/mixed reality if given a predictable 
environment to operate within and potential 
positive psychology impacts and overall wellness. 


Recommendations 


Those who create augmented/mixed reality 
experiences need to clearly define the purpose of 
the designed reality. Users who interact with this 
reality should specifically “opt in” to agree to their 
immersion in the reality. And during the delivery 
of the experience, the reality and reactions of 
those interacting need to be auditable against the 
initial agreed purpose. 


Further Resource 


* Wassom, B. Augmented Reality Law, Privacy, 
and Ethics: Law, Society, and Emerging 
AR Technologies. Waltham, MA: Syngress/ 
Elsevier, 2015. 
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Issue: 


An MR/A/IS environment could 
fail to take into account the 
neurodiversity of the population. 


Background 


Different brains process information differently, 
and MR/A/IS design assumptions could 
potentially limit the value of MR/A/IS experiences 
for many potential users. At the same time, 

an MR/A/IS environment that accommodated 
neurodiversity could be a tool of immense 
potential good. Different people learn differently, 
and a neurodiversity-aware MR/A/IS could 

adapt itself for each individual's strengths and 
preferences. Different brains might well want to 
augment the world differently — for example, 
augmentation for emotional cueing of autistic 
persons. In addition, such an environment would 
offer the opportunity to learn from the ways that 
others experience the world due to different 
cognitive architectures. 


Candidate Recommendations 

Work with MR/A/IS developers to build 
neurodiversity sensitivity into the creation of 
intelligent experiences and hardware. 


Further Resource 


* Metzinger, T., and E. Hildt. Cognitive 
Enhancement. The Oxford Handbook of 
Neuroethics. Oxford, U.K.: Oxford University 
Press, 2011. 
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Section 2 — Mental Health 


While there are proven benefits for creating 
empathy in users or treating PTSD for soldiers 
while utilizing mixed, virtual, or augmented reality, 
there are also potential negative unintended 
consequences via loss of agency, consent, or 
confusion about one’s place in one’s world(s) 
depending on how these tools are used in 
regards to a person suffering from mental health 
issues, or for any individual unused to these 
environments. 





Issue: 


How can Al-enhanced mixed 
reality explore the connections 
between the physical and the 
psychological, the body and 
mind for therapeutic and other 
purposes? What are the risks for 
when an Al-based mixed-reality 
system presents stimuli that 

a user can interact with in an 
embodied, experiential activity? 
Can such MR experiences 
influence and/or control the 
senses or the mind in a fashion 
that is detrimental and enduring? 
What are the short- and long- 
term effects and implications 
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of giving over one’s senses to 
software? Moreover, what are 
the implications for the ethical 
development and use of MR 
applications designed for mental 
health assessment and treatment 
in view of the potential potency 
of this media format compared 
to traditional methodologies? 


Background 


Al-enhanced MR will generate a range of powerful 
applications in healthcare over the next generation, 
from improving medical and surgical outcomes, 

to virtual physicians, to performance visualization 
for athletes. Compelling ultra-high-fidelity systems 
could exploit the brain's neuroplasticity for a variety 
of beneficial (and non-beneficial) ends, including 
present-day treatment of PTSD and anxiety 
disorders using VR. 


Being in a completely mediated VR environment 
could, for example, fool the mind into thinking 
and feeling as it did in an earlier stage of 

one's life, with measurable physiological 

effects. Psychological conditions often have 
accompanying physical ailments that diminish 

or disappear when the psychological condition 
is treated. While the positive impact of MR for 
changing cognition, emotions, and behavior is 
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often talked about as having therapeutic value. If 
one accepts that premise, one has to also accept 
that such changes can occur that have less- 
desirable consequences. 


The converse is true as well. Treating physical 
systems often improves mental states. With 
human augmentation, the physiological and 
psychological can both be automatically 
manipulated or adjusted based on either 
human- or machine-mandated and -controlled 
parameters. In addition to external sensory input, 
we need to consider internal input (implanted 
devices) which deliver information to senses as 
well as deliver medication (or nutrition) based 
upon monitoring emotional or physical states. 


How can mixed reality (MR) be used 
constructively to engage the mind to such 

an extent that physiological mechanisms can 
be controllably affected, and what are the 
ethical implications? We don't have a complete 
understanding of what a human requires to be 
happy and healthy. Does this require interaction 
with the physical world? Or can generated 
experiences be an outlet for those that struggle 
in the real world? Should we always approach 
a user's interaction with a system to help them 
work on real-world problems, or is it okay to let 
them get lost in the generated world? 


A VR system could radically affect how the mind 
processes and synthesizes information, and 
ultimately it could be a way to teach ourselves 
new ways to think and create content. However, 
the long-term effects of immersion are largely 
unknown at this point, and the exploitability of 
a person's (or a larger group's) notion of reality 
raises a host of ethical issues. 
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Creating awareness over who controls what in 
connected systems is critical. Even calling these 
new forms of fiction a series of “realities” blurs 
the line unnecessarily. The idea that there is 
anything human-authored that is “non-fiction” 

is something that needs to be explored on 

a cultural level, or in these ultra-high-fidelity 
systems “truth” will be dictated by an increasingly 
homogeneous and concentrated few. Even if 
these systems are personalized at scale by A/IS, 
fundamental awareness and control need to be 
vested with an individual. 


Questions still need to be answered regarding 
the use of MR as a tool for mental health 
diagnosis and treatment. Thus far, significant 
literature has emerged indicating positive impact 
on mental health and physical functioning using 
theoretically-informed MR applications with 
well-designed content delivered within the more 
controlled (and safe) context of the therapy 
setting, administered and supervised by a well- 
trained clinician. However, what happens if these 
types of VR experiences become commodity 
products that are readily accessible to anyone, 
who might self-diagnose their clinical condition 
and use MR treatment content as “self-help” 
therapy? While some might say this is not much 
different from purchasing a self-help book and 
following the instructions and recommendations 
therein, MR experiences may have a deeper 
impact on a user than reading a book. Similar to 
most areas of mental health care, there is a risk 
that this form of self-diagnosis and treatment 

is based on inaccurate or counterproductive 
information. Another kind of problem may 
emerge if a clinician decides that MR would be 
great for generating a buzz for their practice and 
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result in more business, but hasn't had training 

in its use and safe application. Thus, there are 
issues of concern here from both the patient and 
provider side of the equation. Consequently, we 
need ethical guidelines for the safe and informed 
use of clinical MR applications, much like the way 
that pharmaceutical treatments are managed by a 
well-trained and qualified physician. 


Candidate Recommendation 


Research conducted by qualified mental health 
experts is required in this area to determine how 
people can best approach immersion in new 
realities in ways they can control or mediate 
should potential negative or triggering situations 
take place. 


In the area of clinical practice the American 
Psychological Association's ethical code 
provides a clear and well-endorsed set of 
guidelines that can serve as good starting point 
for understanding and proactively addressing 
some of the issues for the creation and use 

of MR applications (see: www.apa.org/ethics/ 
code/#201e). Three core areas of concerns and 
recommendations can be derived from these 
guidelines (two from the APA code and one 
regarding patient self-help decision-making): 





1. “2.04 Bases for Scientific and 
Professional Judgments 
Psychologists’ work is based upon 
established scientific and professional 
knowledge of the discipline.” 
MR applications that are developed for 
clinical assessment and treatment must be 
based on some theoretical framework and 
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documented with some level of research 
before they can be endorsed as evidence- 
based and promoted to a patient in that 
fashion. In an emerging area like MR, where 
unique and specific guidelines have yet to 
be established, the practitioner must be fully 
transparent about the evidence base for the 
approach and take precautions to preserve 
the safety and integrity of the patient. 


“2.01 Boundaries of Competence 

(a) Psychologists provide services, teach 
and conduct research with populations and 
in areas only within the boundaries of their 
competence, based on their education, 
training, supervised experience, consultation, 
study or professional experience.” 

This one is obvious. MR-delivered mental 
health assessment and treatment may 
require fundamentally different skill sets 
than what is needed for traditional “talk 
therapy” approaches. Clinicians need to 
have specialized training, and possibly in the 
future, some level of certification in the safe 
and ethical use of MR for therapy. 


While not cited as an APA standard, the 
issues regarding patient self-diagnosis 

and self-treatment deserves further 
mention. Mental health conditions can be 
extremely complex and in some instances 
the self-awareness of the patient may be 
compromised. This can oftentimes lead to a 
faulty self-diagnosis as well as the problems 
that arise when the patient searches for 
information via the Internet, where reliable 
and valid content can be questionable. 

The same issues come into play with self- 
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treatment. The problems that can ensue 
are two-fold. 


e The patient makes errors in either or 
both areas and achieves no clinical 
benefit, or worse, aggravates the 
existing condition with an ineffective or 
inappropriate MR approach that actually 
does more harm than good. 


* By pursuing a “seductive” MR self- 
help approach that is misaligned with 
their actual needs or has no evidence 
for its efficacy, the patient could miss 
the opportunity to actually receive 
quality evidence-based care that is 
designed and delivered based on the 
informed judgment of a trained expert 
diagnostician or clinical care provider. 


These two negative impacts could occur if a 
company produces an MR approach without 
sufficient validation and over-promotes or 
markets it to the public as a test or a cure. This 
has been seen over the years with many forms of 
pseudo medicine, and there needs to be some 
principle about the promotion of a MR application 
that has the consumers’ protection in mind. 

This issue is particularly important at the current 
time, in view of all the public exposure, hype, 

and genuine excitement surrounding AR/VR/ 

MR. One can observe new companies emerging 
in the healthcare space without any credible 
expert clinical and/or research guidance. Such 
companies could not only do harm to users, but 
the uninformed development and over-hype of 
the benefits to be derived from a MR clinical 
application leading to negative effects could 
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serve to create the general impression that MR 

is a “snake oil” approach and lead to people not 
seeking (or benefiting from) an otherwise well- 

validated MR approach. 


An example of a “grey area” in this domain 
concerns one of the most common fears that 
people report — public speaking. Technically, in 
an extreme form where it significantly impairs 
social and occupational functioning, public 
speaking anxiety would qualify as a phobia and 
be diagnosed as an anxiety disorder. However, 
since people have some level of sub-clinical fear 
of public speaking that they eventually get over 
with practice, this has been one of the first areas 
where widespread consumer access to public 
speaking VR exposure therapy software has 
occurred . Users can practice their presentation 
“skills” on a low-cost mobile phone driven VR 
HMD (cardboard, Gear VR, Daydream, etc.) in 
front of various types of audiences and settings. 
In this case, most clinicians would not show 
much concern for this type of self-help approach, 
and the potential for damaging effects to a 

user appears to be fairly minimal. But, from this 
example, can we now expect that applications 
will be made readily available for other and 
perhaps more complex anxiety-disorder-based 
phobias (fear of flying, social phobia, driving, 
spiders, intimacy, etc.), or even for PTSD 
treatment? 





From this, general guidelines for the creation, 
distribution, practice methods, and training 
requirements should be established for the 
clinical application of MR for persons with mental 
health conditions. 
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Further Resources 


e Rizzo, A., M. Schultheis, and B. Rothbaum. 
“Ethical Issues for the Use of Virtual Reality in 





the Psychological Sciences” in Ethical Issues 
in Clinical Neuropsychology, edited by S. S. 
Bush, and M. L. Drexler. Lisse, NL: Swets & 
Zeitlinger Publishers, 2002. 





e Wiederhold, B. K., and M. D. Wiederhold. 
Virtual Reality Therapy for Anxiety Disorders: 
Advances in Evaluation and Treatment. 
Washington, DC: American Psychological 
Association, 2005. 








e Botella, C., B. Serrano, R. Baños, and A. 
Garcia-Palacios. “Virtual Reality Exposure- 
Based Therapy for the Treatment of 
Post-Traumatic Stress Disorder: A Review 
of Its Efficacy, the Adequacy of the 
Treatment Protocol, and Its Acceptability.” 
Neuropsychiatric Disease and Treatment 11, 
(2015): 2533-2545. 




















Issue: 


Mixed reality creates 
opportunities for generated 
experiences and high levels of 
user control that may lead certain 
individuals to choose virtual life 
over the physical world. What 

are the clinical implications? 
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Background 


We do not have a complete understanding of 
what a human requires to be happy and healthy. 
Do we require interaction with the physical 
world? Or can generated experiences be an 
outlet for those who struggle in the real world? 
Should we always approach a user's interaction 
with a system to help them work on real-world 
problems, or is it okay to let them get lost in the 
generated world? Some negative examples to 
consider along these lines: 


1. Immersion and escapism could become a 
problem for people who tend to withdraw 
into themselves, become antisocial, and want 
to avoid the real world. This might have to 
be dealt with differently depending on what 
the withdrawal is based on — anxiety, abuse, 
depression, etc. 


2. There will more than likely be issues similar 
to the kind of video-game addictions we see 
now. 


Some positive examples to consider along these 
lines: 


1. AR/VR environments could be used as 
outlets for people who may damage 
themselves, others, or objects in the physical 
world. 


2. AR/VR environments could offer a soothing 
atmosphere for disabled children and adults. 
For example, they could offer experiences 
similar to “stimming” and have relaxing 
music, noises, etc. 


3. There could be an increase of AR/VR 
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therapists and counselors. AR/VR-based Candidate Recommendation 
meditations and mindfulness may also begin 
to proliferate. This could take the form of 
projecting therapists and patients who are 
far apart into the same VR space, projecting 
multiple people into the same VR space for 
meetings, such as Alcoholics Anonymous, 
etc. These methods could be used to help 
people who may not be able to leave the 
home. (For example, therapists have held 


While being conscious to help people avoid 
withdrawal from society where the lack of human 
interaction could increase negative mental health, 
it is important for widespread testing of these 
systems to let these new realities (MR/AR/VR) 
be a tool for exploring interactions to increase 
positive mental health and well-being. 





autism group-counseling sessions inside of Further Resource 

Second Life, reporting that group members e — O'Brolchain, F., T. Jacquemard, D. Monaghan, 
did better expressing themselves when they N. O'Connor, P. Novitzky, and B. Gordijn. “The 
had an avatar with which to participate.) Convergence of Virtual Reality and Social 


Networks: Threats to Privacy and Autonomy.” 
Science Engineering Ethics 22, no. 1 (2016): 
1-29. 
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Section 3 — Education and Training 


There is value in using immersive technologies in 
education and training. That which is experiential 
can provide sustainable training in the long-term. 
Will all senses be stimulated within an immersive 
learning environment? AR/VR could be valuable 
in K-12 classrooms for immersion and interactivity 
with subject material at all different age levels. In 
addition, mixed reality could be one key element 
to lifelong learning and the ability to adapt to 
changing job markets. 





Issue: 


How can we protect worker 
rights and mental well-being 
with the onset of automation- 
oriented, immersive systems? 


Background 


In many workplace environments, humans are 
sharing spaces and tasks with automated systems 
(e.g., robots and/or A/IS algorithms). As these 
relationships increase, there will be increased 
pressure on humans to effectively “team” with 
these systems. There are myriad issues entangled 
in human-machine teaming including A/IS design 
(how do you enable trust?), human-system 
interface (command and control), and enabling 
better situational awareness (sensing and 
understanding). 
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AR/VR/MR will play a large part in these 
solutions, but the art of good immersive 
interfaces and experiences remains largely 
elusive. We currently are in a state where adding 
more data and more sensors is often seen as the 
solution, and yet this does not address the core 
issues of how to increase human performance 
given these information increases. 


Candidate Recommendation 


Two areas need to be considered. First is 
development of the technological capabilities. 
Human factors need to be front-and-center 
throughout the design and testing process, 
particularly with regard not only to efficacy of 
the task execution, but also possible deleterious 
effects on the human, both physical and 
psychological. The second area is implementation 
and deep consideration of the user base. Age, 
psychological state, and other demographic data 
should be considered for use cases, backed by 
research rather than ad hoc determinations. 


Further Resource 


e  Madary, M., and T. K. Metzinger. “Real 
Virtuality: A Code of Ethical Conduct. 
Recommendations for Good Scientific 
Practice and the Consumers of VR- 
Technology.” Frontiers in Robotics 
and Al 3 (February 19, 2016). 
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Issue: 


AR/VR/MR in training/operations 
can be an effective learning 

tool, but will alter workplace 
relationships and the nature 

of work in general. 


Background 


AR/VR/MR is already having an impact in training, 
operations, and production. The capabilities of 
just-in-time knowledge, coaching, and monitoring 
suggests the promise of increased safety and 
productivity. But how will these technologies 
change the workplace, alter career trajectories, 
and impact and influence what, how, and why we 
educate people? 


In addition, the definition of “workplace” 

will radically change. Remote operation and 
increased telepresence capabilities, combined 
with interactive A/IS enabling “always available” 
expertise, make the likelihood high of 
collaborative workspaces that are entirely virtual 
and not necessarily synchronous. While there 
are potential advantages (decreased traffic and 
energy consumption), there will no doubt be 
second- and third-order effects that lead to 
negative outcomes. 


Candidate Recommendation 


Create a task force and living laboratory that 
focuses on the “workplace of the future.” This lab 
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will track emerging technology implementations 
around telepresence and remote collaboration, 
and create test-bed integrations of emerging 
tech, prototyping the “art of the possible,” and 
enabling user studies such that a technologist can 
evaluate, assess, and provide insight into promise 
and pitfalls over the near horizon. 


Further Resource 


e Pellerin, C. “Work: Human-Machine Teaming 
Represents Defense Technology Future.” DoD 
News, Defense Media Activity, Washington, 
DC: Department of Defense, 2015. 











Issue: 


How can we keep the safety and 
development of children and 
minors in mind? 


Background 


AR/VR may be valuable in K-12 classrooms for 
immersion and interactivity with subject material 
at all different age levels. AR can be used to 
interact with shapes, objects, artifacts, models 
of molecules, etc. in a space, while VR can be 
used to explore historical environments, role- 
play in a story or time period, or create a virtual 
whiteboard space for students to collaborate 
and interact in. How can being immersed in a 
different reality interfere with development and 
perception of reality by younger students who 
may not be able to completely differentiate 
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between reality and virtual reality? Would 
escapism and immersion be a problem, for 
example, in mentally ill or unstable teenagers 
who want an escape? How can we protect the 
identity and information of minors, especially if 
virtual experiences might be connected to the 
Internet? 


Candidate Recommendation 


Augment the Consumer Products Safety 
Commission (or equivalent) to include policy/ 
governance over mixed reality products. 
Determine appropriate age restrictions and 
guidelines based on proper research protocols 
and results. 


Further Resource 


e Steinicke, F and G. Bruder. “A Self- 
Experimentation Report About Long-Term 
Use of Fully-Immersive Technology,” 
Proceedings of the 2nd ACM Symposium on 
Spatial User Interaction, (2014): 66-69. 
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Issue: 


Mixed reality will usher 
in a new phase of specialized 
job automation. 


Background 


VR and AR also give rise to a new level of 
automation, where specialized content and 
services, like piano lessons, personalized 
assistance and support, or even tourism guidance 
could be consumed at any given time and place. 
This will bring better customized services into our 
lives at a lower cost and higher availability. It is 
also, however, likely to negatively impact a broad 
class of jobs. 


Candidate Recommendation 


Governments are advised to keep close watch 
over the automation of personalized services 
through mixed-reality technology and offer 
alternative education and training to professionals 
in fields that are expected to be affected. 


Further Resource 





* Stanford University. “Artificial Intelligence and 
Life in 2030: One Hundred Year Study on 
Artificial Intelligence.” Stanford, CA: Stanford 
University, 2016. 
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Issue: 


A combination of mixed reality 
and A/IS will inevitably replace 
many current jobs. How will 
governments adapt policy, and 
how will society change both 
expectations and the nature 
of education and training? 


Background 


It is clear that many current tasks in society 

will move from human-actuated to being 
accomplished by machine and/or algorithm. The 
Industrial Revolution gives an historical taste of 
this type of change, but given the depth and 
breadth of digital penetration into human life, it 
will be an even more profound sea change. There 
are two main areas of immediate concern. First 
is for the population — essentially, “what will | do 
for a living?” Educational and training missions 
will need rethinking, and infrastructure will need 
to be created or leveraged to enable rapid career 
changes and skill acquisition. 


Second, government will need to consider the 
societal ramifications of automation replacing 


@greative 
commons 


human labor, and no doubt policy will need to be 
crafted to enable agility in the workforce along 
with models for how humans work and thrive in 
increasingly virtual environments populated by 
artificial agents. 


Candidate Recommendation 


Create a working group to look at industries and 
job areas most likely to be replaced or heavily 
augmented by a combination of mixed reality and 
Al/loT. Similarly, the group would work to predict 
near-term and longer-term job needs and growth 
areas. Look to leverage the existing community 
college system as a platform for “21st century 
trades,” enabling rapid acquisition of necessary 
skills along with ongoing training. 


Further Resources 


* Nutting, R. “No, ‘Truck Driver’ Isn't the Most 
Common Job in Your State.” MarketWatch, 
February 12, 2015. 








* Stanford University. “Artificial Intelligence and 
Life in 2030: One Hundred Year Study on 
Artificial Intelligence.” Stanford, CA: Stanford 
University, 2016. 











* Stern, A. Raising the Floor: How a Universal 
Basic Income Can Renew Our Economy and 
Rebuild the American Dream. New York: 
PublicAffairs 2016. 
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Section 4 — The Arts 


Throughout history, the arts have been a 

means for human expression and often healthy 
escapism, as well as for social and political 
commentary. The imminent arrival of culturally 
pervasive mixed-reality technologies has the 
potential to dramatically impact and permanently 
alter the methods and tools by which artists earn 
their living. With this in mind, how can humanity 
best approach the interdisciplinary and cross- 
cultural impacts that the new AR/VR artistic 
paradigms will offer? 





Issue: 


There is the possibility of 
commercial actors to create 
pervasive AR/VR environments 
that will be prioritized in user's 
eyes/vision/experience. 


Background 


In the near future, users will filter their digital 
landscapes by opting in or opting out of mixed- 
reality information-delivery mechanisms driven 
by A/IS frameworks that will both structure and, 
in many cases, alter or curate the data for private, 
opaque ends. 
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With specific regard to AR, how will the digital 
public landscape not simply be absorbed 

by private commercial interests, but allow 

virtual space for citizens and artists to freely 
participate? Will artistic content be algorithmically 
subordinated to commercial content? 


Candidate Recommendation 


Provide users/citizens the option to always “opt 
out” of any immersive environment to which 

they may be exposed and provide transparency 
and consent options to make this possible. 

This transparency could include not only the 
constituent algorithms, but also information about 
the identity of private actors behind the data. 





Issue: 


There is the possibility that AR/ 
VR realities could copy/emulate/ 
hijack creative authorship and 
intellectual and creative property 
with regard to both human 
and/or Al-created works. 


Background 


There exists the possibility for certain types of art 
forms or certain creative ideas when expressed 

in this new modality to be algorithmically 
suppressed. How can we make sure there is even 
distribution and access to ideas? 
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Mixed reality presents unique opportunities for 
developers, artists, and story-tellers to both build 
upon and challenge existing modes of content 
creation, while helping to forge original tools 

and methodologies in the realization of new 
artistic media. Virtual reality (VR) and 360 video 
borrow narrative and artistic techniques from 


virtual environments that reflect original rights or 
ownership to validate, recognize, and remunerate 
artists for original work. In addition to research, 
new forms of copyright will surely need to be 
conceived and codified that are more appropriate 
for the highly collaborative, inter-media, and 
virtual environments within which many of these 


their gaming, theater, cinema and architecture 
antecedents; however, these media also present 
new occasions for developers to fashion novel 
modes of editing, point of view (POV), and sound 
(for example). 


Using many of the same creative tools, AR 
provides a way to use public spaces as a canvas 
for meaningful cultural exchange and, in doing 
so, affords the user a fresh way of seeing such 
spaces as a more open and democratic media 
environment. The creative community writ large 
can leverage AR as an instrument of new media 
content creation, public media production, and 
artistic expression, which could result in a freer, 
more effective use of public space, as well as 

a more imaginative exchange of ideas between 
citizens. Finally, A/IS frameworks used to 
generate artworks are becoming more accessible, 
which raises questions of the role of the human 
artist and ethical issues of authorship and creative 
rights. The philosophical debate around the 
concepts “author” and “artist” with regard to 
created works is not a new one in the humanities 
or the legal world. However, these concepts take 
on entirely new dimensions when infusing the 
discussion with the role of a non-human actor in 
the creative process. 


Candidate Recommendation 


Research methods to allow new forms of creative 
copyright to be embedded within physical and 
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mixed reality works will be created. 


Further Resources 


Cartiere C., and M. Zebracki, eds., The 
Everyday Practice of Public Art: Art, Space, 
and Social Inclusion. New York: Routledge, 
2016. 


Geroimenko, V. Augmented Reality Art: From 
an Emerging Technology to a Novel Creative 
Medium. Cham, Switzerland: Springer, 2014. 


Foucault, M. “Space, Knowledge and Power,” 
in The Foucault Reader edited by P. Rabinow. 
Harmondsworth, U.K.: Penguin, 1984. 


Baudrillard, J. Simulacra et Simulation. 
Translated by P. Foss, P. Patton, and P. 
Beitchman. New York: Semiotext(e), 1983. 


Morey, S., and J. Tinnell, eds. Augmented 
Reality: Innovative Perspectives across Art, 
Industry, and Academia. Anderson, SC: Parlor 
Press, 2016. 


Lanier, J. Dawn of the New Everything: 
Encounters with Reality and Virtual Reality, 
New York: Henry Holt, and Co., 2017. 








Grau, O. Virtual Art: From Illusion to 
Immersion, Cambridge, MA: The MIT Press, 
2003. 
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Section 5 — Privacy Access and Control 


While concerns over personal data access abound 
within existing Internet or loT environments, the 
nature of the imminent pervasive and immersive 
landscapes of mixed reality provides unique new 
challenges regarding the nature of user identity 
and control. 





Issue: 


Data collection and control issues 
within mixed realities combined 
with A/IS present multiple ethical 
and legal challenges that ought 
to be addressed before these 
realities pervade society. 


Background 


AR’s and VR's potential for persistent, ubiquitous 
recording could undermine the reasonable 
expectation of privacy that undergirds privacy- 
law doctrine as expressed in constitutional law, 
tort, and statute (Roesner et al., 2014). Like other 
emerging technologies, it may force society to 
rethink notions of privacy in public. Furthermore, 
the mobility of AR devices in particular 
exacerbates challenges to privacy in private 
spaces, such as the home, that have traditionally 
been subject to the strongest privacy protections. 
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Ubiquitous recording will challenge expectations 
of privacy both in public and private spaces. 
Excessive storage and data logging will inevitably 
create a target for law enforcement (think the 
Alexa case). The personalized consumption of 
controversial immersive content could pose 
challenges for effective public oversight and 
erode the distinction between what is real 

and what is permissible. The ability of A/IS 
paired with AR to match disparate data sets will 
challenge a bystander’s ability to control her/his 
public image. 


This also prompts the question of data 
ownership, access, and control in VR and AR. 

If users divulge personal or identifying data, we 
should have clear assurances that their virtual 
and physical identities can and will be protected 
within such virtual worlds. This also applies to 
accidental collection of data by VR systems to 
better customize the technology. It is important 
to question the level of control we have over our 
data and privacy when integrating these pervasive 
technologies into our lives. 


Further, mixed-reality applications must be 
secured against tampering. As technology 
mediates the way users view their surroundings, 
cybersecurity is vital to ensure that only they can 
see the information on their displays. Unsecured 
applications not only leave data vulnerable, 

but create the possibility of digital assault 

or false light. 
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Also, as AR platforms become the gateway to 
certain pieces of information, developers should 
consider the discriminatory effects of placing 
information behind that gateway — especially 
since the display of incomplete information is a 
form of misuse that can lead to discrimination. If 
some vital piece of information is only available 
via AR, or only available to a particular AR 
sandbox, some people will inevitably be locked 
out of that information (of course, this criticism 
could apply to any communications technology, 
so the solution may be opportunities for public 
access [e.g., libraries] rather than design). 


Consider a mixed-reality scenario in which a user 
“sees” a photorealistic avatar commit a crime 
(a real crime, whether in simulation or not) but 
the avatar depicts (is cloaked) as an altogether 
different person (or persons) than the person 
who is “seen” by third-party witnesses. In that 
case, only an identity-management system 

will know who the true perpetrator was. What 
will happen under such circumstances to the 
1) perpetrators of the crime (what constitutes 
probable cause and reasonable search?) and 
2) what happens to the person whose identity 
was “falsely used” within mixed reality? What if 
a person is falsely accused because immersed 
witnesses have “seen” them commit a crime? 
What access to identity- management software 
does each of these constituencies have? 


Candidate Recommendation 


Further research is required in assessing the 
implications of data collection, A/IS, and mixed 
reality to include benefits and definition of 
boundaries. 
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Further Resource 


* Stanford University. “Artificial Intelligence and 
Life in 2030: One Hundred Year Study on 
Artificial Intelligence.” Stanford, CA: Stanford 
University, 2016. 














Issue: 


Like other emerging 
technologies, AR/VR will force 
society to rethink notions of 
privacy in public and may 
require new laws or regulations 
regarding data ownership in 
these environments. 


Background 


If a user has a specific interaction with mixed- 
reality avatars, can and should this particular 
storyline development become proprietary? If 
users divulge personal or identifying data, we 
should have clear assurances that their virtual 
and physical identities can and will be protected 
within such virtual worlds. This also applies to 
accidental collection of data by VR systems to 
better customize the technology. It is important 
to question the level of control we have over our 
data and privacy when integrating these pervasive 
technologies into our lives. 


Facial recognition and other machine learning 
applications that can match disparate data sets 
will hamper people's ability to control their own 
image. For example, an AR application that 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 238 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Mixed Reality in Information and Communications Technology Committee 





matches publicly available information with facial 
recognition will strip bystanders of anonymity 
without their consent (Denning, Dehlawi, and 
Kohno 2014). 


The development of specialized content for VR 
— e.g., violent shooting games, highly sexualized 
or illicit content — limits public oversight of 
controversial content consumption. Considering 
AR provides a high level of immersion, mixed 
reality will challenge established policy and social 
norms around privacy and data control. 


Candidate Recommendation 


Further research is needed on data-control 
issues and an A/IS or mixed-reality “guardian” or 
“agent” will be required to identify any potentially 
negative environments or issues within those 
environments based on an individual's preset 
requirements regarding data and identity issues. 
Including the potential role of blockchain may 
be part of this study. Further, it is incumbent 
upon technologists to educate the public on 

the benefits and potential for abuse of A/IS 

and mixed reality. 


Further Resources 


e IEEE P7006™, Standard for Personal Data 
Artificial Intelligence (Al) Agent. 








* — Stanford University. “Artificial Intelligence and 





Life in 2030: One Hundred Year Study on 
Artificial Intelligence.” Stanford, CA: Stanford 
University, 2016. 
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Issue: 


Users of Al-informed mixed- 
reality systems need to 
understand the known effects 
and consequences of using those 
systems in order to trust them. 


Background 


Trust will be essential to the widespread adoption 
of A/IS that pervades our lives and helps make 
increasingly crucial decisions. With black boxes 
playing influential roles, trust will be difficult to 
earn. Openness and transparency could be ideal 
principles to guide the development of intelligent 
mixed reality in a way that would alleviate much 
understandable wariness. Trust is a factor in 

not only the corporate use of personal data, 

but also in A/IS algorithms and the increasingly 
compelling mixed-reality illusions superimposed 
on the physical world. In a world where one’s 
very perception has been delegated to software, 
unprecedented levels of trust in systems 

and data — and openness and transparency — 
will be needed to ensure the technology's 
responsible progress. 


Candidate Recommendations 


Establish a new kind of user guide for MR/A/ 

IS focused on transparency and end-user 
understanding of the constituent components. 
Users should be able to understand the systems 
and their logic if they are going to opt-in in an 
informed manner. Perhaps there is a place for 

a neutral, trusted, and independent third party 
to evaluate MR/A/IS products and experiences 
along these lines. 
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Prioritizing ethical and responsible artificial intelligence has become a widespread goal for 
society. Important issues of transparency, accountability, algorithmic bias, and others are 
being directly addressed in the design and implementation of autonomous and intelligent 
systems (A/IS). While this is an encouraging trend, a key question still facing technologists, 
manufacturers, and policy makers alike is, what should be the specific metrics of societal 
success for “ethical Al” once it's being used? 


For A/IS to demonstrably advance the well-being of humanity, there needs to be concise 
and useful indicators to measure those advancements. However, there is not a common 
understanding of what well-being indicators are, or which ones are available. Technologists 
will use best-practice metrics available even if, unbeknownst to them, said metrics are 
inappropriate or, worse, potentially harmful. To avoid unintended negative consequences 
and to increase value for users and society, clear guidance on what well-being is and how 
it should be measured is needed. 


Common metrics of success include profit, gross domestic product (GDP), consumption 
levels, occupational safety, and economic growth. While important, these metrics fail to 
encompass the full spectrum of well-being for individuals or society. Psychological, social, 
and environmental factors matter. Where these factors are not given equal priority to fiscal 
metrics of success, technologists risk causing or contributing to negative and irreversible 
harms to our planet and population. 


This document identifies examples of existing well-being metrics that capture such 
factors, allowing the benefits of A/IS to be more comprehensively evaluated. While these 
indicators vary in their scope and use, they expand the focus of impact to aspects of 
human well-being that are not currently measured in the realms of A/IS. 


When properly utilized, these metrics could provide an opportunity to test and monitor 
A/IS for unintended negative consequences that could diminish human well-being. 
Conversely, these metrics could help identify where A/IS would increase human well-being, 
providing new routes to societal and technological innovation. By corollary, A/IS can also 
increase the measurement and efficiency of well-being indicators. 
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This Committee, along with the IEEE P7010™ Standard Working Group, Well-being Metrics 
Standard for Ethical Artificial Intelligence and Autonomous Systems, was created with the 
belief that A/IS should prioritize human well-being as an outcome in all system designs, 
using the best available and widely accepted well-being metrics as their reference point. 





This document is divided into the following sections: 


e An Introduction to Well-being Metrics (What you need to know) 





* The Value of Well-being Metrics for A/IS (Why you should care) 








e Adaptation of Well-being Metrics for A/IS (What you can do) 


Appendix: 


The following sections are included in the Appendix as separate documents to provide 
readers with an introduction to existing individual and societal level well-being metrics 
currently in use: 


e The State of Well-being Metrics. This section identifies well-being metrics being used 
today by social scientists, international institutions, and governments to provide an 
overall introduction to well-being. 


e The Happiness Screening Tool for Business Product Decisions. This tool is provided 
as an example of how well-being indicators can inform decisions. 





Disclaimer: While we have provided recommendations in this document, it should be understood these do not represent a 
position or the views of IEEE but the informed opinions of Committee members providing insights designed to provide expert 
directional guidance regarding A/IS. In no event shall IEEE or IEEE-SA Industry Connections Activity Members be liable for any 
errors or omissions, direct or otherwise, however caused, arising in any way out of the use of this work, regardless of whether 
such damage was foreseeable. 
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Section 1 — An Introduction 
to Well-being Metrics 


and Development (OECD) Guidelines on 
Measuring Subjective Well-being (p. 12). This 
holistic definition of well-being encompasses 
individual, social, economic, and governmental 
circumstances as well as human rights, 
capabilities, environmental protection, and fair 
labor, as these circumstances and many others 
form the basis for human well-being. 


This section provides a brief overview of what 
well-being metrics are outside of the context of 
A/IS to provide a background for readers who 
may not be familiar with these areas. 











Issue: 


There is ample and robust science 
behind well-being metrics and 
use by international and national 1. 
institutions, yet many people 

in the A/IS field and corporate 
communities are unaware that 
well-being metrics exist, or what 
entities are using them. 


Well-being metrics fall into four categories: 
Subjective or survey-based indicators 


* Survey-based or subjective well-being (SWB) 
indicators are being used by international 
institutions and countries to understand 
levels of reported well-being within a country 
and for aspects of citizen demographics. 
Examples include the European Social Survey, 
Bhutan’s Gross National Happiness Indicators, 
and well-being surveys created by The UK 
Office for National Statistics. Survey-based 
or subjective metrics are also employed in 
the field of positive psychology and in the 
World Happiness Report, and the data are 








Background 





The concept of well-being refers to an evaluation 
of the general goodness of a state or event 
to the individual or community as a distinct 





moral or legal evaluation. The term itself has 
been used and defined in various ways across 
different contexts and fields. For the purposes 
of this committee, well-being is defined as 
encompassing human satisfaction with life and 
the conditions of life, flourishing (eudaimonia), 
and positive and negative affect, following 

the Organization for Economic Cooperation 
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employed by researchers to understand the 
causes, consequences, and correlates of 
well-being as subjects see it. The findings 

of these researchers provide crucial and 
necessary guidance to policy makers, leaders, 
and others in making decisions regarding 
people's subjective sense of well-being. 
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2. Objective indicators 


* Objective well-being indicators have been 
used to understand conditions enabling 
well-being of countries and to measure the 
impact of companies. They are in used by 
organizations like the OECD with their Better 
Life Index (which also includes subjective 
indicators), and United Nations with their 
Millennium Development Goal Indicators. 
For business, the Global Reporting Initiative, 
SDG Compass, and B-Corp provide broad 
indicator sets. 














3. Composite indicators (indices that 
aggregate multiple metrics) 


e Aggregate metrics combine subjective and/ 


or objective metrics to produce one measure. 


Examples of this are the UN's Human 
Development Index, the Social Progress 
Index, and the United Kingdom's Office 
of National Statistics Measures of National 


Well-being. 














4. Social media sourced data 


* Social media is source used to measure 
the well-being of a geographic region or 
demographics, based on sentiment analysis 
of publicly available data. Examples include 
the Hedonometer and the World Well-being 
Project. 





The appendix The State of Well-being Metrics 
provides a broad primer on the state of well- 
being metrics. 
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Candidate Recommendation 


A/IS policy makers and manufacturers 
(including academics, designers, engineers, 

and corporate employees) should prioritize 
having all their stakeholders learn about well- 
being metrics as potential determinants for how 
they create, deploy, market, and monitor their 
technologies. This process can be expedited 

by having organizations including the Global 
Reporting Initiative (GRI), B-Corp, and Standards 
Development Organizations (SDO) create 
certifications, guidelines, and standards that 
demonstrate the value of holistic, well-being- 
centric reporting guidelines for the A/IS public 
and private sectors. 


Further Resources 


e The IEEE P7010™ Standards Working Group, 
Wellbeing Metrics Standard for Ethical 
Artificial Intelligence and Autonomous 
Systems has been formed with the aim of 
identifying well-being metrics for applicability 
to A/IS today and in the future. All are 
welcome to join the working group. 








e On 11 April 2017, IEEE hosted a dinner 
debate at the European Parliament in 
Brussels to discuss how the world's top 
metric of value (gross domestic product) 
must move Beyond GDP to holistically 
measure how intelligent and autonomous 
systems can hinder or improve human 
well-being: 





e Prioritizing Human Well-being in the Age 
of Artificial Intelligence (Report) 








e Prioritizing Human Well-being in the Age 
of Artificial Intelligence (Video) 
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Section 2 — The Value of Well-being 


Metrics for A/IS 


Well-being metrics, in the form of triple-bottom 
line benefits (“people, planet, and profit”) for 
the corporate world, and in the form of tools 

to measure a population's well-being for policy 
makers, can provide value to A/IS technologists. 
Where technologists may be unaware of how 
systems could negatively impact human well- 
being, by increasing awareness of common 
indicators and their designed intent, they can 
avoid harm while increasing benefit. 


In addition, a key value for the use of well-being 
metrics for A/IS technologists comes in the 
form of predictive modeling (forecasting 
outcomes based on data analysis and 
probabilities), either for unintended 
consequences, or as a unique means 

of innovation regarding metrics or areas 

of consideration not currently being 

measured today. 





Issue: 


Many people in the A/IS field 

and corporate communities are 
not aware of the value well-being 
metrics offer. 
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Background 


While many organizations are aware of the need 
to incorporate sustainability measures as part of 
their efforts, the reality of bottom line, quarterly 
driven shareholder growth is a traditional 

metric prioritized within society at large. Where 
organizations exist in a larger societal ecosystem 
equating exponential growth with success, as 
mirrored by GDP or similar financial metrics, 
these companies will remain under pressure 

to deliver results that do not fully incorporate 
societal and environmental measures and goals 
along with existing financial imperatives. 


Along with an increased awareness of how 
incorporating sustainability measures beyond 
compliance can benefit the positive association 
with an organization's brand in the public sphere, 
by prioritizing the increase of holistic well-being, 
companies are also recognizing where they can 
save or make money and increase innovation 

in the process. 


For instance, where a companion robot outfitted 
to measure the emotion of seniors in assisted 
living situations might be launched with a typical 
“move fast and break things” technological 
manufacturing model, prioritizing largely fiscal 
metrics of success, these devices might fail in 

the market because of limited adoption. However, 
where they also factor in data aligning with 
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uniform metrics measuring emotion, depression, 
or other factors (including life satisfaction, affect, 
and purpose), the device might score very high 
on a well-being scale comparable to the Net- 
Promoter Score widely used today. If the device 
could significantly lower depression according 

to metrics from a trusted source like the World 
Health Organization, academic institutions testing 
early versions of systems would be more able to 
attain needed funding to advance an A/IS well- 
being study overall. While these are hypothetical 
scenarios, they are designed to demonstrate 

the value of linking A/IS design to well-being 
indicators where possible. 





This is a key point regarding the work of this 
Committee — rather than focus on the negative 
aspects of how A/IS could harm humans, the 
implementation of uniform well-being metrics 
will help provably demonstrate how these 
technologies can have a positive influence on 
society. 


The good news in regards to this subject is 

that thought leaders in the corporate arena 

have recognized this multifaceted need to 

utilize metrics beyond fiscal indicators. In 2013, 
PricewaterhouseCoopers released a report called 
Total Impact Approach: What CEOs Think from 
PricewaterhouseCoopers: (where total impact 
refers to a “holistic view of social, environmental, 
fiscal and economic dimensions”) where they 
noted: 








187 CEOs across the globe shared their 
views on the value of measuring total 
impact. From all industries, they explored 
the benefits, opportunities and challenges 


of a total impact approach. There's an 
overwhelming consensus (85% CEOs) that 
results from a total impact approach would 
be more insightful than financial analysis 
alone. Business leaders saw the more holistic 
perspective useful in not only managing 
their business, but also in communicating 
with certain stakeholders. But less than 25% 
of CEOs measure their total impact with 

the lack of availability of data or a robust 
framework holding them back. 


This report, along with more recent work being 
done by other thought-leading organizations in 
the public sector like the OECD in their February, 
2017 Workshop, Measuring Business Impacts 

on People’s Well-Being, demonstrates the desire 
for business leaders to incorporate metrics of 
success beyond fiscal indicators for their efforts. 
The B-Corporation movement has even created a 
new legal status for “a new type of company that 
uses the power of business to solve social and 
environmental problems.” Focusing on increasing 
“stakeholder” value versus shareholder returns 
alone, forward-thinking B-Corps are building trust 
and defining their brands by provably aligning 
their efforts to holistic metrics of well-being. 











From a mental health perspective, well-being is 
also important to business. Happy workers are 
more productive than employees who are not 
engaged in their careers. There are also fewer 
issues with absenteeism: people miss work less 
and have fewer health claims. 
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Candidate Recommendation 


A/IS and well-being experts should work directly 
with the business community to identify existing 
metrics or combinations of indicators that would 
bring the greatest value to businesses focused on 
the “triple bottom line” (accounting for economic, 
social, and environmental impacts) increase of 
human well-being. (Noting, however that well- 
being metrics should only be used with consent, 
respect for privacy, and with strict standards for 
collection and use of these data). 


In addition, any stakeholders creating A/IS in 

the business or academic, engineering, or policy 
arenas are advised to review the Appendix listing 
well-being metrics to familiarize themselves with 

existing indicators already relevant to their work. 


Further Resources 


* PwC. Total Impact Approach: What CEOs 
Think. 





e World Economic Forum. The Inclusive Growth 





and Development Report. January 16, 2017. 
Geneva, Switzerland: World Economic Forum. 
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Issue: 

By leveraging existing work in 
computational sustainability or 
using existing indicators to model 
unintended consequences of 
specific systems or applications, 
well-being could be better 
understood and increased 

by the A/IS community and 
society at large. 


Background 


To date, there does not exist a definitive well- 
being metric that encompasses every aspect of 
individual and societal well-being that could serve 
as a common metric like the GDP for all A/IS 
manufacturers. Moreover, data may or may not 
exist within the context one wishes to measure 
or improve. 


Modeling for Unintended Consequences 


There is a potential for synergy when adapting 
well-being indicators for the use of A/ 

IS. This potential is in avoiding unintended 
consequences. Two challenges to face when 
exploring this potential are: (1) Identifying which 
indicators to select to model potential unintended 
consequences; and, (2) Understanding how to 
predict unintended consequences when data are 
lacking or are incomplete. 
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Machine-learning and other tools have the ability 
to map out potential consequences with greater 
specificity and efficiency than humans. In this 
way, A/IS could be utilized to map out potential 
consequences regarding how products, services, 
or systems might affect end users or stakeholders 
in regards to specific well-being indicators. In 

this way, models could be run during the design 
phase of a system, product, or service to predict 
how it could improve or potentially harm end 
users, analogous to human rights assessments 
provided by the United Nations Guiding Principles 
Reporting Framework. 


As the exchange of A/IS related data regarding 
an individual (via personalized algorithms, in 
conjunction with affective sensors measuring 
and influencing emotion, etc.) and society (large 
data sets representing aggregate individual 
subjective and objective data) is widely available 
via establishing tracking methodologies, this data 
should be classified to match existing well-being 
indicators so devices or systems can be provably 
aligned to the increase of human well-being 
(satisfaction with life and the conditions of life, 
positive affect, and eudaimonic well-being). 


As an example, today popular robots like Pepper 
are equipped to share data regarding their usage 
and interaction with humans to the cloud. This 
allows almost instantaneous innovation, as 

once an action is validated as useful for one 
Pepper robot, all other units (and ostensibly their 
owners) benefit as well. As long as this data 
exchange happens via pre-determined consent 
with their owners, this “innovation in real-time” 
model can be emulated for the large-scale 
aggregation of information relating to existing 
well-being metrics. 


A crucial distinction between well-being metrics 
and potential interventions in their use is 

that a well-being metric does not dictate an 
intervention, but points the way for developing 
an intervention that will push a metric in a 
positive direction. For example, a team seeking 
to increase the well-being of people using 
wheelchairs found that when provided the 
opportunity to use a smart wheelchair, some 
users were delighted with the opportunity 

for more mobility, while others felt it would 
decrease their opportunities for social contact 
and lead to an overall decrease in their well- 
being. The point being that even increased 
well-being due to a smart wheelchair does not 
mean that this wheelchair should automatically 
be adopted. Well-being is only one value in the 
mix for adoption, where other values to consider 
would be human rights, respect, privacy, justice, 
freedom, culture, etc. 





Computational Sustainability 


Computational sustainability is an area of study 
within the A/IS community that demonstrates 
that the A/IS community is already showing 
interest in well-being even when not using 

this term, as the concept of sustainability 
encompasses aspects of well-being. 





Computational sustainability directly relates to the 
use of these technologies to increase social good 
in ways that could be uniquely tied to existing 
well-being metrics. As defined by The Institute 

of Computational Sustainability, the field is 
designed to provide “computational models for a 
sustainable environment, economy, and society” 
and their project summary notes that: 
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Humanity’s use of Earth's resources is threatening 
our planet and the livelihood of future 
generations. Computing and information science 
can — and should — play a key role in increasing 
the efficiency and effectiveness in the way we 
manage and allocate our natural resources. 

We propose an expedition in Computational 
Sustainability, encompassing computational 

and mathematical methods for a sustainable 
environment, economy, and society. 


AAAI, (the Association for the Advancement of 
Artificial Intelligence) the world’s largest global 
body dedicated to the advancement of artificial 
intelligence had a special track on computational 
sustainability at their 2017 conference. The 
description of the track provides helpful specifics 
demonstrating the direct alignment between the 
work of this Committee and the A/IS community 
at large: 





This special track invites research papers 

on novel concepts, models, algorithms, 

and systems that address problems in 
computational sustainability. We are looking 
for a broad range of papers ranging 

from formal analysis to applied research. 
Examples include papers explaining how the 
research addresses specific computational 
problems, opportunities, or issues underlying 
sustainability challenges and papers 
describing a sustainability challenge or 
application that can be tackled using 

Al methods. Papers proposing general 
challenges and data sets for computational 
sustainability are also welcome. All Al topics 
that can address computational sustainability 
issues are appropriate, including machine 
learning, optimization, vision, and robotics, 


and others. Sustainability domains include 
natural resources, climate, and the 
environment (for example, climate change, 
atmosphere, water, oceans, forest, land, 
soil, biodiversity, species), economics and 
human behavior (for example, human well- 
being, poverty, infectious diseases, over- 
population, resource harvesting), energy 
(for example, renewable energy, smart grid, 
material discovery for fuel cell technology) 
and human-built systems (for example, 
transportation systems, cities, buildings, 
data centers, food systems, agriculture). 


Candidate Recommendations 


e Work with influencers and decision-makers 
in the computational sustainability field to 
cross-pollinate efforts of computational 
sustainability in the A/IS field and the 
well-being communities to expedite efforts 
to identify, align, and advance robust and 
uniform indicators into current models that 
prioritize and increase human well-being. 
Develop cross-pollination between the 
computational sustainability and well-being 
professionals to ensure integration of well- 
being into computational sustainability, and 
vice-versa. 


e — Explore successful programs like LEED 
Building Design Standards, ISO 2600 
Corporate Responsibility, ISO 37101 
Sustainable Development Standards, and 
others to determine what new standards 
or certification models along these 
lines approach would be valuable and 
operationalizable for A/S. 
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Further Resources 


* Gomes, C. P. “Computational Sustainability: 
Computational Methods for a Sustainable 
Environment, Economy, and Society” in The 
Bridge: Linking Engineering and Society. 
Washington, DC: National Academy of 
Engineering of the National Academies, 
2009. 











e Meadows, D. H., D. L. Meadows, J. Randers, 
and W. W. Behrens, Ill. The Limits to Growth. 





New York: Universe Books, 1972. Reissued 
in 2004 by Chelsea Green Publishing & 
Earthscan. 


e LEED Building Design Standards program. 





e ISO 2600, Guidance on Social Responsibility. 





e ISO 37101, Sustainable Development in 
Communities 
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Issue: 


Well-being indicators provide 

an opportunity for modeling 
scenarios and impacts that could 
improve the ability of A/IS to 
frame specific societal benefits 
for their use. 


Background: 


There is a lack of easily available or widely 
recognized scenarios along these lines. 


Candidate Recommendation 


Rigorously created well-being assessments could 
be utilized as a public “scoreboard,” or statement 
of intent, that would provide innovation 
opportunities for technologists as well as a form 
of public accountability for human sustainability. 


Further Resources 


The following schema and well-being assessment 
tool provide an initial attempt to visualize how 
A/IS technologists can utilize well-being metrics 
in their work. By modeling the potential positive 
or negative impacts of technologies across a full 
spectrum of financial, environmental, and social 
impacts (e.g., a “triple bottom line” well-being 
indicator model) A/IS technologists can better 
avoid negative unintended consequences for 
human well-being, while increasing innovation 
and positive human well-being for their work. 
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Schema of A/IS and the Stakeholders Involved in Their Development 


Observations of well-being 
metrics + possible rewards 


Best well-being metrics available in 
each case. 


Best well-being metrics available in 
each case. 


Well-being metrics, 
SDG Index, HDI, happiness, custom. 


Best product and service well-being 
metrics available. 


Organization well-being metrics, SDG 
Compass business indicators, SDG 
Index government indicators. 


Country, State, City well-being 
metrics, HDI, SOG Compass business 
indicators, SOG Index government 
indicators. 
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Agents and candidate goals 


Stakeholders involved in A/IS development 
educated, trained, and empowered to 
prioritize well-being ethical considerations so 
that these technologies are advanced for the 
benefit of humanity. 

Goal: Develop A/IS that achieve well-being 
impact goals for its users and stakeholders. 


A/IS Systems 
Goal: Achieve well-being goals and 
sub-objectives within constraints presented. 


People 
Goal: Achieve well-being, SDGs, HDi, custom. 


Products and Services 
Including A/IS. 
Goal: Achieve product well-being impact. 


Organizations 

Including A/IS. 

Goal: Achieve organizational well-being 
impact. 


Countries, States, Cities, Planet, etc. 
Goal: Achieve well-being for their citizens, 
society and planet, SDGs, HDI, custom. 








Schema of A/IS systems and the stakeholders involved in their development, adapting and operationalizing well-being metrics for ethical A/IS in a world model. 


The schema represents a model of the world where the stakeholders (designers, engineers, 
technologists, researchers, managers, users, etc.) involved in A/IS development adapt and operationalize 
well-being metrics for ethical A/IS. Stakeholders can visualize important entities in the world as agents 
with different goals that receive observations and possible rewards from the environment and make 
actions that could have positive and negative impacts to the well-being of different agents. 


This schema could help to assess, in different cases, the well-being metrics that the A/IS should take 
into account and the well-being metrics of the impacts that A/IS actions could and can cause, related to 
important elements in the world like: people, products, organizations, climate, countries, etc. An applied 
case of this schema could be seen in the following well-being impact assessment. 
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Well-being Impact Assessment 


Here is a concept for simple A/IS well-being impact assessment, based on Maslow's Hierarchy of Need 
(where the Hierarchy would be considered an accredited and contextually appropriate metric of use). 
Given that a working definition of well-being including both individual and societal key performance 
indicators (KPIs) is still being developed, this metric is general and used for illustrative purposes only. 





Please also note that this is a purely conceptual framework used as a directional teaching tool for 
readers. It doesn’t yet include an evaluative component or reflect the holistic nature of well-being at this 
time like The Happiness Screening Tool (based on the government of Bhutan’s Policy Screening Tool) 
provided in the Appendix. It should be noted that any impact assessment created by A/IS and well- 
being experts working together identify best-in-class (existing) metrics within specific contexts of use. 





Individual Individual Environment Individual Social Social 
Direct Indirect Direct Indirect Direct Indirect 


eae a E 


Esteem 


Indicators: 





nil impact = 0 negative impact = — positive impact = + unknown impact = ? 
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The following examples are provided to demonstrate specific A/IS applications within this framework 
and include: a retail kiosk robot, a small factory arm, a mental health chatbot, and a companion robot. 
The goal of these diagrams is to provide a sample of how the work of matching established well-being 
metrics to A/IS work could progress. 


Retail Kiosk Robot Individual Individual Environment Individual Social Social 


Direct Indirect Direct Indirect Direct Indirect 
Safety 


Self-Actualization 


E ee A ee eee 
ETER ese 
ie |e ee 
ee eee ee 





Using this tool, the retail kiosk robot scores are mildly beneficial in the category of Individual Direct (i.e., 
reduced barriers to goal attainment) and Environmental Direct (i.e. use of resources), while strongly 
beneficial in Social Direct (i.e., better access to mental health support), but mildly unbeneficial in 
Environment Indirect (i.e., carbon footprint), and unknown in Social Indirect (i.e., job loss) categories. 
The robot is “helpful and kind,” but of limited utility or interaction value. Another example of a negative 
impact on well-being is gendering, racial identification, or physical attributes of kiosk robots (such as a 
slim, youthful appearing, Caucasian, female), leading to harmful stereotyping. 
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Small Factory Arm | Individual Individual Environment Individual Social Social 


Direct Indirect Direct Indirect Direct Indirect 
Ss a a 
ee 
a i ee a 
m am ts | 
po foo |e [>| 


Self-Actualization 


The tool indicates that robots need to be assessed more thoroughly on their safe operations to better 
answer impact assessment, and that this is also a robot with very limited interaction with people. But the 
diagram shows how the arm could have a potentially negative impact on self-worth and belonging, but a 
positive impact on basic needs both for individuals and society. 





Mental Health Individual Individual Environment Individual Social Social 
Chatbot Direct Indirect Indirect Direct Indirect 


Basic Needs 


Safety 


Self-Actualization C 
Overall Impact Strong + 
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There is evidence that a mental health aide chatbot could improve individual self esteem and ultimately 
reduce self harm, but there is little evidence supporting claims that this would improve society directly 
or indirectly. The reliance on artificial support may have a net negative impact on society. However, 

this would need to be determined by the A/IS and well-being experts applying this methodology once 
created in a robust and rigorous manner. 


Companion Robot | Individual Individual Environment Individual Social Social 
like Paro Direct Indirect Direct Indirect Direct Indirect 


Overall Impact 





For a small resource cost, a companion robot can provide significant psychological assistance. On the 
one hand, this makes society more caring, but on the other hand reliance on artificial companionship 
shows a lack of social resources in this area. A potential negative impact is development of reliance on 
companionship and negative impact on people who lose access to companion robot. 


The Happiness Project Screening Tool for Business provided in the Appendix could also augment this if 
a product shows a low or negative score in the areas of well-being. Another set of metrics that could be 
used in a more detailed schema are the Kingdom of Bhutan’s nine domains of well-being: psychological 
well-being, health, community vitality, living standards, governance, environment diversity, culture, 
education, and time use. 





Whatever established well-being metrics that may be utilized for such a methodology, it is critical 
for A/IS technologists and well-being experts to work in unison to create assessment tools using best 
in class data, indicators, and practices in their potential analysis and use. 
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Section 3 — Adaptation of Well-being 


Metrics for A/IS 


This section focuses on areas of immediate 
attention for A/IS technologists to be aware 
of regarding well-being metrics in an effort to 
aid their work and avoid negative unintended 
consequences. 





Issue: 


How can creators of A/IS 
incorporate measures of 
well-being into their systems? 


Background 


Just as undirected A/IS can lead to negative 
outcomes, A/IS directed only to specific ends 
without considering human well-being can lead 
to negative side effects. Without practical ways 
of incorporating widely shared ways of measuring 
and promoting well-being metrics and expected 
well-being outcomes available to designers, A/IS 
will likely lack beneficence. 


Once well-being metrics are widely recognized 
as a directional requirement for society, 
conceptually, one would like such measures 

to be supported by the engines of change and 
leverage within society. A/IS will be an increasing 
portion of such engines. How might designers 
architect systems to include such measures as 
considerations while executing their primary 


objectives? How will these measures be adapted 
as we learn more? 


Existing metrics of well-being could be formulated 
into a sub-objective of the A/IS. In order to 
operate with respect to such sub-objectives, it 

is instrumental to evaluate the consequences 

of the A/IS's actions. As practical systems are 
bounded and can predict over only limited 
horizons, it may be necessary to supplement 
these evaluations with both biases toward virtues 
and deontological guidelines or soft constraints 
as lesser supplemental components, informed 
by the well-being metrics and their precursors 

or constituents. 


As these well-being sub-objectives will be only 
a subset of the intended goals of the system, 
the architecture will need to balance multiple 
objectives. Each of these sub-objectives may 

be expressed as a goal, or as a set of rules, or as 
a set of values, or as a set of preferences, and 
those can be combined as well, using 
established methodologies from intelligent 
systems engineering. 


For example, people, organizations, and 

A/IS, collaborating together, could understand the 
well-being impacts and objectives of products, 
services, organizations, and A/IS within the 
context of the well-being of communities, cities, 
countries, and the planet using the SDG Index 
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and Dashboards, the SDG Compass Inventory 

of Business Indicators and other metrics. This 
collaboration of people, organizations and 

A/IS could make decisions and take actions with 
high expected utility to well-being objectives and 
goals such as those stated in the Sustainable 
Development Goals and similar institutions. This 
collaboration could lead to a more humane, 
organizational, and computational sustainability 
for individuals, all of society, and the planet. 

















International organizations, lawmakers, and 

policy experts can specify core values and/or 
sub-objectives as rules for the benefit of society 
utilizing well-being metrics as a starting point 

and these can be pluggable and hierarchical by 
jurisdiction. Similarly, industry organizations would 
be able to specialize norms and industry self- 
regulation (e.g., any automated flight attendants 
should prevent onboard smoking and sit down 
during takeoff) as a layer. 


System designers should ensure situational 
awareness as well as prediction of the 
consequences of their actions based on 

some world model. They could also layer in 
their own sub-objectives and make the system's 
values explicit. 


Resellers, service organizations, or owners that 
have particular primary goals for their systems 
would still be able to specify primary goals for the 
system (e.g., mowing lawns, doing taxes, etc.), 
and those would be alongside the other deeper- 
down subgoals and values as well for societal 
benefit, public safety, etc., directly relating 

to established well-being metrics. 


End users would have the opportunity to layer 
on their own preferences in these systems, and 
would also be able to get an explanation and 
inventory of the types of objectives or value 
systems the A/IS holds relating to established 
well-being metrics, including what permissioning 
is required for modifying or removing them. 


Candidate Recommendation 


Formation of a working group to develop a 
blueprint for the fluid and evolving (institutional 
learning) operationalization of A/IS well-being 
indicators for the various stakeholders (e.g., 
technicians, coders, and system designers), 
international well-being oriented organizations, 
lawmakers, and policy experts, industry 
organizations, retailers, resellers, service 
organizations and owners, and end users. 


Candidate Recommendation 


Creation of technical standards for representing 
dimensions, metrics, and evaluation guidelines 
for well-being metrics and their precursors and 
constituents within A/IS. This would include 
ontologies for representing requirements as well 
as a testing framework for validating adherence 
to well-being metrics and ethical principles. 

(For more information, please see IEEE P7010™ 
Standards Working Group mentioned above). 
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Further Resources 


e Calvo, R. A., and D. Peters. Positive 
Computing: Technology for Well-Being and 
Human Potential. Cambridge MA: MIT Press, 
2014 








* Collette Y., and P. Slarry. Multiobjective 
Optimization: Principles and Case Studies 
(Decision Engineering Series). Berlin, 
Germany: Springer, 2004. doi: 10.1007/978- 
3-662-08883-8. 








* Greene, J. et al. “Embedding Ethical Principles 
in Collective Decision Support Systems,” in: 
Proceedings of the Thirtieth AAAI Conference 
on Artificial Intelligence, 4147—4151. Palo 
Alto, CA: AAAI Press, 2016. 





* Li, L. et al. “An Ontology of Preference-Based 
Multiobjective Evolutionary Algorithms,” 2016. 
CoRR abs/1609.08082. 








e A. FT Winfield, C. Blum, and W. Liu. 
“Towards an Ethical Robot: Internal Models, 
Consequences and Ethical Action Selection,” 
in Advances in Autonomous Robotics 
Systems. Springer, 2014, pp. 85—96. 








e Gershman, S. J., E. J. Horvitz, and J. B. 
Tenenbaum. “Computational rationality: A 








converging paradigm for intelligence in brains, 
minds, and machines.” Science 349, no. 
6245 (2015): 273-278. 





e PositiveSociallmpact: Empowering people, 
organizations and planet with information 
and knowledge to make a positive impact 
to sustainable development. 
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Issue: 


A/IS technologies designed to 
replicate human tasks, behavior, 
or emotion have the potential 
to either increase or decrease 
well-being. 


Background 


A/IS are already being executed in ways that 
could dramatically increase human well-being 
or, possibly, have an undue coercive effect on 
humans. 


A/IS technologies present great opportunity 

for positive change in every aspect of 

society. However, sophisticated manipulative 
technologies utilizing A/IS can also restrict the 
fundamental freedom of human choice, and 

are able to manipulate humans who consume 
customized content without recognizing the 
extent of manipulation. Software platforms are 
moving from targeting content to much more 
powerful and potentially harmful “persuasive 
computing.” A/IS with sophisticated manipulation 
technologies (so-called “big nudging”) will 

be able to guide individuals through entire 
courses of action, whether it be a complex work 
process, consumption of free content, or political 
persuasion. 


There is also a related concern that big nudging 
can be done without anyone realizing harm 
is occurring. With deep learning methods, 


This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License. 257 


The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 


Well-being 


technologies may not be understood, much less 
contemplated. This begs the age-old question: 
just because one can do something, does that 
mean one should? Hence, there is a need to 
understand A/IS well-being related processes 
and impacts further, and to devise ways to 
protect people from harm and secure 
well-being in the furtherance of A/IS. 


A/IS may also deceive and harm humans 

by posing as humans. With the increased 

ability of artificial systems to meet the Turing 
test (an intelligence test for a computer that 
allows a human to distinguish human from 
artificial intelligence), there is a significant risk 
that unscrupulous operators will abuse the 
technology for unethical commercial, or outright 
criminal, purposes. The widespread manipulation 
of humans by A/IS and loss of human free 
agency, autonomy, and other aspects of human 
flourishing, is by definition a reduction in 
human well-being. Without taking action to 
prevent it, it is highly conceivable that A/IS will 
be used to deceive humans by pretending to be 
another human being in a plethora of situations 
or via multiple mediums. 


Without laws preventing A/IS from simulating 
humans for purposes like deception and 
coercion, and enforcing A/IS to clearly identify 
as such, mistaken identity could also reasonably 
be expected. 
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Candidate Recommendation 


To avoid potential negative unintended 
consequences, A/IS manufacturers, and society 
in general, should prioritize the analysis and 
implementation of practices and policy that 
secures or increases human well-being, including: 


e Well-being metrics to guide the development 
and implementation of A/IS should increase 
human well-being, defined subjectively in 
terms of cognitive, affective, and eudaimonic 
domains, and objectively in terms of 
conditions enabling well-being. 


e — While individuals may enjoy the ability of A/IS 
to simulate humans in situations where they 
are pure entertainment, explicit permission 
and consent by users in the use of these 
systems is recommended, and the well- 
being impacts on users should be monitored, 
researched, and considered by the A/IS 
community in an effort to provide services 
and goods that improve well-being. As part 
of this, it is important to include multiple 
stakeholders, including minorities, the 
marginalized, and those often without 
power or a voice. 


e The implications of A/IS on human well- 
being are important issues to research and 
understand. A literature review to determine 
the status of academic research on the 
issue of A/IS impacts on human well-being 
needs to be conducted and aggregated 
in a centralized repository for the A/IS 
community. 
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Further Resources 


e Helbing, D. et al. “Will Democracy Survive 
Big Data and Artificial Intelligence?” Scientific 
American, February 25, 2017._ 








¢ Schenker, J. L. “Can We Balance Human 
Ethics with Artificial Intelligence?” 
Techonomy, January 23, 2017. 








* Bulman, M. “EU to Vote on Declaring Robots 
To Be ‘Electronic Persons.” Independent, 
January 14, 2017. 








e Nevejan, N. for the European Parliament. 
“European Civil Law Rules in Robotics.” 
October 2016. 





- “The Al That Pretends To Be Human,” 
LessWrong blog post, February 2, 2016. 





e — Chan, C. “Monkeys Grieve When Their Robot 
Friend Dies.” Gizmodo, January 11, 2017. 








Issue: 


Human rights law is sometimes 
conflated with human well-being, 
leading to a concern that a focus 
on human well-being will lead 

to a situation that minimizes the 
protection of inalienable human 
rights, or lowers the standard 

of existing legal human rights 
guidelines for non-state actors. 
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Background 


International human rights law has been firmly 
established for decades and the protection of 
human rights must be an end result in itself. 
Some countries or regimes have highlighted 

the use or increase of certain “well-being” 
measures as justification to violate human rights, 
as happens in countries that conduct ethnic 
cleansing or mistreat refugees or immigrants 
who are portrayed as threatening a nation’s 
culture or economic structure. 


While the use of well-being metrics to justify 
human rights violations is an unconscionable 
perversion of the nature of any well-being 

metric, these same practices happen today in 
relation to the GDP. For instance, today, according 
to the International Labor Organization (ILO) 
approximately 21 million people are victims of 
forced labor (slavery) representing between 9% 
to 56% of various countries current GDP income. 
These clear human rights violations, from sex 
trafficking and child armies, to indentured farming 
or manufacturing labor, increase a country’s GDP. 





Well-being metrics and mechanisms should 

also take into consideration, and happen in 
conjunction with, independent assessments on 
respect and international obligations to promote, 
protect, and fulfill a full spectrum of human 
rights. For example, the use of the goal of well- 
being in the context of repairing and enhancing 
humans, predictive policing, or autonomous 
weapons systems to protect the public may have 
negative impacts on the rights of individuals or 
groups. Moreover, the development and delivery 
of A/IS should adopt a human rights approach 
to technology, including, but not limited to, the_ 
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UN Guiding Principles on Human Rights (also 
known as the Ruggie principles). 





To avoid issues of conflation and confusion, it 
is critical to note the following: human rights 
involves adhering to the firmly established 
application of international human rights law. 
Well-being metrics are designed to measure 
the efficacy of the implementation of 
methodologies and policy related to 
individual and societal flourishing. 


Well-being as a value is also distinct from 

justice, responsibility, and freedom. But A/IS 
technologies can be narrowly conceived from 

an ethical standpoint and still be legal and safe 
in their usage following existing practices, but not 
contribute to human well-being. In this regard, 
well-being considerations do not displace other 
issues of human rights or ethical methodologies, 
but rather complement them. 


Candidate Recommendation 


Human rights and human well-being should not 
be held as trade-offs, with one to be prioritized 
over the other. In this regard, well-being metrics 
can be complementary to the goals of human 
rights, but cannot and should not be used as 

a proxy for human rights or any existing law. 


Further Resources 


e Project Include - The site features an open 
source manual for creating diversity in tech 
and highlights three key points for creating 
change: inclusion, comprehensiveness, and 
accountability. 
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e OpenDiversityOrg initiative from Double 
Union and Project Include have an action 
document with a lot of recommendations. 


+ “The Diversity Debt” by Susan Wu at Project 
Include is a compelling example of converting 
a problem into innovation language. 








Issue: 


A/IS represents opportunities 
for stewardship and restoration 
of natural systems and securing 
access to nature for humans, but 
could be used instead to distract 
attention and divert innovation 
until the planetary ecological 
condition is beyond repair. 


Background 


Human well-being, the existence of many other 
species, as well as economic and social systems, 
draw from and depend upon healthy ecological 
systems and a healthy local and planetary 
environment. Research using geo-data finds that 
human well-being is enhanced through access 
to nature. Many bank on technology to answer 
the threats of climate change, water scarcity, soil 
degradation, species extinction, deforestation, 
deterioration of biodiversity, and destruction 

of ecosystems that threaten humankind and 
other life forms. 
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While technology may be the answer for some 
of these threats, it is unclear whether benefits 
extend beyond those from high socio-economic 
class to the majority of people, particularly the 
middle class and working poor, as well as those 
suffering from abject poverty, fleeing disaster 
zones or otherwise lacking the resources to 
meet their needs. For example, in cities in China 
where air pollution is so prevalent that the air is 
unhealthy, a few schools have covered “outdoor” 
fields with domes full of purified air while most 
children must risk their lungs when playing 
outside, or play indoors. Moreover, it is well- 
understood that ecological crises, such as sea_ 
level rise and fisheries depletion, will not only 
negatively impact business interests, but it will 
have a significantly more devastating impact on 
the poor and developing nations than the wealthy 
and developed nations. 








Candidate Recommendation 


Well-being metrics employed for A/IS should 
include measures for ecological/environmental 
sustainability that point the direction toward 
stewardship and restoration of natural systems 
and ensure equitable environmental justice. 


Candidate Recommendation 


Convene a committee to issue findings on the 
modalities and potentials already identified in 
which A/IS makes progress toward stewardship 
and restoration of natural systems; trends in 
the A/IS field that represent threats to and 
opportunities for ecological sustainability and 
environmental justice; and areas for suggested 
future innovation and implementation. 


Further Resources 


e Newton, J. “Well-being and the Natural 
Environment: An Overview of the Evidence.” 
August 20, 2007. 








* Dasgupta, P. Human Well-Being and the 
Natural Environment. Oxford, U.K.: Oxford 
University Press, 2001. 








* Haines-Young, R., and M. Potschin. “The 
Links Between Biodiversity, Ecosystem 
Services and Human Well-Being,” in 
Ecosystem Ecology: A New Synthesis, edited 
by D. Raffaelli, and C. Frid. Cambridge, U.K.: 
Cambridge University Press, 2010. 








* Hart, S. Capitalism at the Crossroads: Next 
Generation Business Strategies for a Post- 
Crisis World. Upper Saddle River, NJ: Pearson 
Education, 2010. 








e United Nations Department of Economic 
and Social Affairs. “Call for New Technologies 





to Avoid Ecological Destruction.” Geneva, 
Switzerland, July 5, 2011. 





e Pope Francis. Encyclical Letter Laudato Si, 
On the Care of Our Common Home. 
May 24, 2015. 
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Issue: 


The well-being impacts of A/IS 
applied to human genomes are 
not well understood. 


Background 


As A/IS are increasingly used to interpret the 
health significance of our genomics data ("deep 
genomics”) and to contribute to the subsequent 
engineering and editing of our genomes, 
important ethical and governance questions 

are in the background that provide an opportunity 
to utilize well-being metrics to ensure the 
beneficial development of genomic research 

as it relates to A/IS. 


Imagine this scenario: 


6 A.M., Washington, DC — Erika wakes 
up and quickly checks her “digital DNA 
avatar,” a digital version of her genetic 
blueprint as it evolves day by day. 

The avatar knows a lot about her as 

it constantly monitors the interactions 
between her genes, analyzes her bodily 
fluids and diet, as well as integrates data 
about the air quality around her. Her 
avatar proposes a few advices about 
food choices and exercise patterns. 
Everything seems in check, nothing 

to be worried about. For now. 


A first overarching reflection concerns the 
relationship between well-being and an 


increasing ability to understand and engineer our 
genomes: How do in-depth and personalized 
understanding of how our genomes function 
and evolve relate to the notion of well-being 

as measured traditionally and/or according to 
well-being measures? When does a reductionist 
interpretation of the health significance of our 
genomics data threaten our well-being? 


Other significant questions include: 


e How accurate will the predictive health 
data coming from the convergence of 
A/IS and genomics be? 


e How will these health predictions be used, 
and who will have access to them? 


e Do pharmaceutical and insurance 
companies have the right to use and profit 
from your health data predictions/modeling 
without giving you any benefits back in 
return? 


* Would it threaten your self-worth if those 
handling your health data know a lot 
of biological details about your body? 


e Is it ethical for a prospective employer to 
ask how your health will look like in the 
next decade? 


Answers to these questions are not easy to 
capture, but their impact on well-being within 
society is profound. 


The convergence of genomics technologies and 





A/IS creates new opportunities to define our 
identity and well-being within a simple narrative 
in which our genes have the power to tell us who 
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and how well we are. As A/IS are increasingly 
used to interpret the health significance of 

our genomics data (“deep genomics”) and to 
contribute to the subsequent engineering/editing 
of our genomes, we should consider important 
ethical and governance questions. 


There is an urgent need to concurrently discuss 
how the convergence of A/IS and genomic 

data interpretation will challenge the purpose 
and content of relevant legislation that preserve 
well-being, such as, for the United States, the 
Health Insurance Portability and Accountability 
Act (HIPAA) and the Genetic Information Non- 
Discrimination Act (GINA). Finding the right 
balance of protection and regulation in using 
A/IS to interpret the health significance of 
genomics data will be important. Too much 
regulation could endanger precision medicine 
initiatives in some countries, while others would 
be leading the bio-race. Too little regulation could 
leave citizens vulnerable to different forms of 
threats to their well-being. 
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Candidate Recommendation 


A working committee should be convened 
gathering those at the sharp end of genomics, 
A/IS, ethics, and governance to start a 
conversation with different communities to 
better understand the impact on well-being 
of the use of A/IS to interpret (and engineer) 
genomics data. 


Candidate Recommendation 


Relevant expert and legislative committees 
should commission a study on the impact on 
well-being of deep genomics, meaning at the 
convergence of genomics and A/IS. Such a study 
is recommended to encompass diverse fields 

of expertise in philosophy, sociology, ethics, 
biosafety, biosecurity, and genomics governance. 
Recommendations from the study should draft 
proposals to frame debates in legislatures and 
help lawmakers start developing appropriate 
legislation to govern A/IS applied to genomes 
for the well-being of society. 
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WASHINGTON, D.C. 20502 


October 12, 2016 


Dear colleagues: 


Advances in Artificial Intelligence (AI) technology have opened up new markets and new opportunities 
for progress in critical areas such as health, education, energy, and the environment. In recent years, 
machines have surpassed humans in the performance of certain specific tasks, such as some aspects of 
image recognition. Experts forecast that rapid progress in the field of specialized artificial intelligence 
will continue. Although it is very unlikely that machines will exhibit broadly-applicable intelligence 
comparable to or exceeding that of humans in the next 20 years, it is to be expected that machines will 
reach and exceed human performance on more and more tasks. 


As a contribution toward preparing the United States for a future in which AI plays a growing role, this 
report surveys the current state of AI, its existing and potential applications, and the questions that are 
raised for society and public policy by progress in AI. The report also makes recommendations for 
specific further actions by Federal agencies and other actors. A companion document lays out a strategic 
plan for Federally-funded research and development in AI. Additionally, in the coming months, the 
Administration will release a follow-on report exploring in greater depth the effect of Al-driven 
automation on jobs and the economy. 


The report was developed by the NSTC’s Subcommittee on Machine Learning and Artificial Intelligence, 
which was chartered in May 2016 to foster interagency coordination, to provide technical and policy 
advice on topics related to AI, and to monitor the development of AI technologies across industry, the 
research community, and the Federal Government. The report was reviewed by the NSTC Committee on 
Technology, which concurred with its contents. The report follows a series of public-outreach activities 
spearheaded by the White House Office of Science and Technology Policy (OSTP) in 2016, which 
included five public workshops co-hosted with universities and other associations that are referenced in 
this report. 


OSTP also published a Request for Information (RFI) in June 2016, which received 161 responses. The 
submitted comments were published by OSTP on September 6, 2016. Consistent with the role of Big 
Data as an enabler of AI, this report builds on three previous Administration reports on Big Data 
referenced in this report. 


In the coming years, AI will continue to contribute to economic growth and will be a valuable tool for 
improving the world, as long as industry, civil society, and government work together to develop the 
positive aspects of the technology, manage its risks and challenges, and ensure that everyone has the 
opportunity to help in building an Al-enhanced society and to participate in its benefits. 
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Executive Summary 


As a contribution toward preparing the United States for a future in which Artificial Intelligence (AI) 
plays a growing role, we survey the current state of AI, its existing and potential applications, and the 
questions that are raised for society and public policy by progress in AI. We also make recommendations 
for specific further actions by Federal agencies and other actors. A companion document called the 
National Artificial Intelligence Research and Development Strategic Plan lays out a strategic plan for 
Federally-funded research and development in AI. 


Applications of AI for Public Good 


One area of great optimism about AI and machine learning is their potential to improve people’s lives by 
helping to solve some of the world’s greatest challenges and inefficiencies. Many have compared the 
promise of AI to the transformative impacts of advancements in mobile computing. Public- and private- 
sector investments in basic and applied R&D on AI have already begun reaping major benefits to the 
public in fields as diverse as health care, transportation, the environment, criminal justice, and economic 
inclusion. The effectiveness of government itself is being increased as agencies build their capacity to use 
AI to carry out their missions more quickly, responsively, and efficiently. 


AI and Regulation 


AI has applications in many products, such as cars and aircraft, which are subject to regulation designed 
to protect the public from harm and ensure fairness in economic competition. How will the incorporation 
of AI into these products affect the relevant regulatory approaches? In general, the approach to regulation 
of Al-enabled products to protect public safety should be informed by assessment of the aspects of risk 
that the addition of AI may reduce alongside the aspects of risk that it may increase. If a risk falls within 
the bounds of an existing regulatory regime, moreover, the policy discussion should start by considering 
whether the existing regulations already adequately address the risk, or whether they need to be adapted 
to the addition of AI. Also, where regulatory responses to the addition of AI threaten to increase the cost 
of compliance, or slow the development or adoption of beneficial innovations, policymakers should 
consider how those responses could be adjusted to lower costs and barriers to innovation without 
adversely impacting safety or market fairness. 


Currently relevant examples of the regulatory challenges that Al-enabled products present are found in 
the cases of automated vehicles (AVs, such as self-driving cars) and Al-equipped unmanned aircraft 
systems (UAS, or “drones”). In the long run, AVs will likely save many lives by reducing driver error and 
increasing personal mobility, and UAS will offer many economic benefits. Yet public safety must be 
protected as these technologies are tested and begin to mature. The Department of Transportation (DOT) 
is using an approach to evolving the relevant regulations that is based on building expertise in the 
Department, creating safe spaces and test beds for experimentation, and working with industry and civil 
society to evolve performance-based regulations that will enable more uses as evidence of safe operation 
accumulates. 


Research and Workforce 


Government also has an important role to play in the advancement of AI through research and 
development and the growth of a skilled, diverse workforce. A separate strategic plan for Federally- 
funded AI research and development is being released in conjunction with this report. The plan discusses 
the role of Federal R&D, identifies areas of opportunity, and recommends ways to coordinate R&D to 
maximize benefit and build a highly-trained workforce. 
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Given the strategic importance of AI, moreover, it is appropriate for the Federal Government to monitor 
developments in the field worldwide in order to get early warning of important changes arising elsewhere 
in case these require changes in U.S. policy. 


The rapid growth of AI has dramatically increased the need for people with relevant skills to support and 
advance the field. An Al-enabled world demands a data-literate citizenry that is able to read, use, 
interpret, and communicate about data, and participate in policy debates about matters affected by AI. AI 
knowledge and education are increasingly emphasized in Federal Science, Technology, Engineering, and 
Mathematics (STEM) education programs. AI education is also a component of Computer Science for 
All, the President’s initiative to empower all American students from kindergarten through high school to 
learn computer science and be equipped with the computational thinking skills they need in a technology- 
driven world. 


Economic Impacts of AI 


AT’s central economic effect in the short term will be the automation of tasks that could not be automated 
before. This will likely increase productivity and create wealth, but it may also affect particular types of 
jobs in different ways, reducing demand for certain skills that can be automated while increasing demand 
for other skills that are complementary to AI. Analysis by the White House Council of Economic 
Advisors (CEA) suggests that the negative effect of automation will be greatest on lower-wage jobs, and 
that there is a risk that Al-driven automation will increase the wage gap between less-educated and more- 
educated workers, potentially increasing economic inequality. Public policy can address these risks, 
ensuring that workers are retrained and able to succeed in occupations that are complementary to, rather 
than competing with, automation. Public policy can also ensure that the economic benefits created by AI 
are shared broadly, and assure that AI responsibly ushers in a new age in the global economy. 


Fairness, Safety, and Governance 


As AI technologies move toward broader deployment, technical experts, policy analysts, and ethicists 
have raised concerns about unintended consequences of widespread adoption. Use of AI to make 
consequential decisions about people, often replacing decisions made by human-driven bureaucratic 
processes, leads to concerns about how to ensure justice, fairness, and accountability—the same concerns 
voiced previously in the Administration’s Big Data: Seizing Opportunities, Preserving Values report of 
2014,' as well as the Report to the President on Big Data and Privacy: A Technological Perspective 
published by the President’s Council of Advisors on Science and Technology in 2014.” Transparency 
concerns focus not only on the data and algorithms involved, but also on the potential to have some form 
of explanation for any Al-based determination. Yet AI experts have cautioned that there are inherent 
challenges in trying to understand and predict the behavior of advanced AI systems. 


Use of AI to control physical-world equipment leads to concerns about safety, especially as systems are 
exposed to the full complexity of the human environment. A major challenge in AI safety is building 
systems that can safely transition from the “closed world” of the laboratory into the outside “open world” 
where unpredictable things can happen. Adapting gracefully to unforeseen situations is difficult yet 
necessary for safe operation. Experience in building other types of safety-critical systems and 
infrastructure, such as aircraft, power plants, bridges, and vehicles, has much to teach AI practitioners 


' “Big Data: Seizing Opportunities, Preserving Values,” Executive Office of the President, May 2014, 
https://www. whitehouse. gov/sites/default/files/docs/big_data_privacy_report_may 1 2014.pdf. 





2 The President’s Council of Advisors on Science and Technology, “Report to the President: Big Data and Privacy: 
A Technological Perspective,” Executive Office of the President, May 2014, 
https://www.whitehouse.gov/sites/default/files/microsites/ostp/PCAST/pcast_big data_and_privacy_- 
_may_2014.pdf. 
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about verification and validation, how to build a safety case for a technology, how to manage risk, and 
how to communicate with stakeholders about risk. 


At a technical level, the challenges of fairness and safety are related. In both cases, practitioners strive to 
avoid unintended behavior, and to generate the evidence needed to give stakeholders justified confidence 
that unintended failures are unlikely. 


Ethical training for AI practitioners and students is a necessary part of the solution. Ideally, every student 
learning AI, computer science, or data science would be exposed to curriculum and discussion on related 
ethics and security topics. However, ethics alone is not sufficient. Ethics can help practitioners understand 
their responsibilities to all stakeholders, but ethical training should be augmented with technical tools and 
methods for putting good intentions into practice by doing the technical work needed to prevent 
unacceptable outcomes. 


Global Considerations and Security 


AI poses policy questions across a range of areas in international relations and security. AI has been a 
topic of interest in recent international discussions as countries, multilateral institutions, and other 
stakeholders have begun to access the benefits and challenges of AI. Dialogue and cooperation between 
these entities could help advance AI R&D and harness AI for good, while also addressing shared 
challenges. 


Today’s AI has important applications in cybersecurity, and is expected to play an increasing role for both 
defensive and offensive cyber measures. Currently, designing and operating secure systems requires 
significant time and attention from experts. Automating this expert work partially or entirely may increase 
security across a much broader range of systems and applications at dramatically lower cost, and could 
increase the agility of the Nation’s cyber-defenses. Using AI may help maintain the rapid response 
required to detect and react to the landscape of evolving threats. 


Challenging issues are raised by the potential use of AI in weapon systems. The United States has 
incorporated autonomy in certain weapon systems for decades, allowing for greater precision in the use of 
weapons and safer, more humane military operations. Nonetheless, moving away from direct human 
control of weapon systems involves some risks and can raise legal and ethical questions. 


The key to incorporating autonomous and semi-autonomous weapon systems into American defense 
planning is to ensure that U.S. Government entities are always acting in accordance with international 
humanitarian law, taking appropriate steps to control proliferation, and working with partners and Allies 
to develop standards related to the development and use of such weapon systems. The United States has 
actively participated in ongoing international discussion on Lethal Autonomous Weapon Systems, and 
anticipates continued robust international discussion of these potential weapon systems. Agencies across 
the U.S. Government are working to develop a single, government-wide policy, consistent with 
international humanitarian law, on autonomous and semi-autonomous weapons. 


Preparing for the Future 


AI holds the potential to be a major driver of economic growth and social progress, if industry, civil 
society, government, and the public work together to support development of the technology with 
thoughtful attention to its potential and to managing its risks. 


The U.S. Government has several roles to play. It can convene conversations about important issues and 
help to set the agenda for public debate. It can monitor the safety and fairness of applications as they 
develop, and adapt regulatory frameworks to encourage innovation while protecting the public. It can 
provide public policy tools to ensure that disruption in the means and methods of work enabled by AI 
increases productivity while avoiding negative economic consequences for certain sectors of the 
workforce. It can support basic research and the application of AI to public good. It can support 
development of a skilled, diverse workforce. And government can use AI itself to serve the public faster, 
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more effectively, and at lower cost. Many areas of public policy, from education and the economic safety 
net, to defense, environmental preservation, and criminal justice, will see new opportunities and new 
challenges driven by the continued progress of AI. The U.S. Government must continue to build its 
capacity to understand and adapt to these changes. 


As the technology of AI continues to develop, practitioners must ensure that Al-enabled systems are 
governable; that they are open, transparent, and understandable; that they can work effectively with 
people; and that their operation will remain consistent with human values and aspirations. Researchers 
and practitioners have increased their attention to these challenges, and should continue to focus on them. 


Developing and studying machine intelligence can help us better understand and appreciate our human 
intelligence. Used thoughtfully, AI can augment our intelligence, helping us chart a better and wiser path 
forward. 


A full list of the recommendations made in this report is on page 40. 
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Introduction 


Artificial Intelligence (AT) has the potential to help address some of the biggest challenges that society 
faces. Smart vehicles may save hundreds of thousands of lives every year worldwide, and increase 
mobility for the elderly and those with disabilities. Smart buildings may save energy and reduce carbon 
emissions. Precision medicine may extend life and increase quality of life. Smarter government may serve 
citizens more quickly and precisely, better protect those at risk, and save money. Al-enhanced education 
may help teachers give every child an education that opens doors to a secure and fulfilling life. These are 
just a few of the potential benefits if the technology is developed with an eye to its benefits and with 
careful consideration of its risks and challenges. 


The United States has been at the forefront of foundational research in AI, primarily supported for most of 
the field’s history by Federal research funding and work at government laboratories. The Federal 
Government’s support for unclassified AI R&D is managed through the Networking and Information 
Technology Research and Development (NITRD) program, and supported primarily by the Defense 
Advanced Research Projects Agency (DARPA), the National Science Foundation (NSF), the National 
Institutes of Health (NIH), the Office of Naval Research (ONR), and the Intelligence Advanced Research 
Projects Activity (IARPA). Major national research efforts such as the National Strategic Computing 
Initiative, the Big Data Initiative, and the Brain Research through Advancing Innovative 
Neurotechnologies (BRAIN) Initiative also contribute indirectly to the progress of AI research. The 
current and projected benefits of AI technology are large, adding to the Nation’s economic vitality and to 
the productivity and well-being of its people. A companion document lays out a strategic plan for 
Federally-funded research and development in AI. 


As a contribution toward preparing the United States for a future in which AI plays a growing role, we 
survey the current state of AI, its existing and potential applications, and the questions that progress in AI 
raise for society and public policy. We also make recommendations for specific further actions by Federal 
agencies and other actors. 


A Brief History of AI 


Endowing computers with human-like intelligence has been a dream of computer experts since the dawn 
of electronic computing. Although the term “Artificial Intelligence” was not coined until 1956, the roots 
of the field go back to at least the 1940s,° and the idea of AI was crystalized in Alan Turing’s famous 
1950 paper, “Computing Machinery and Intelligence.” Turing’s paper posed the question: “Can machines 
think?” It also proposed a test for answering that question,’ and raised the possibility that a machine might 
be programmed to learn from experience much as a young child does. 


In the ensuing decades, the field of AI went through ups and downs as some AI research problems proved 
more difficult than anticipated and others proved insurmountable with the technologies of the time. It 
wasn’t until the late 1990s that research progress in AI began to accelerate, as researchers focused more 
on sub-problems of AI and the application of AI to real-world problems such as image recognition and 
medical diagnosis. An early milestone was the 1997 victory of IBM’s chess-playing computer Deep Blue 


3 See, e.g., Warren S. McCulloch and Walter H. Pitts, “A Logical Calculus of the Ideas Immanent in Nervous 
Activity,” Bulletin of Mathematical Biophysics, 5:115-133, 1943. 


4 Restated in modern terms, the “Turing Test” puts a human judge in a text-based chat room with either another 
person or a computer. The human judge can interrogate the other party and carry on a conversation, and then the 
judge is asked to guess whether the other party is a person or a computer. If a computer can consistently fool human 
judges in this game, then the computer is deemed to be exhibiting intelligence. 
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over world champion Garry Kasparov. Other significant breakthroughs included DARPA’s Cognitive 
Agent that Learns and Organizes (CALO), which led to Apple Inc.’s Siri; IBM’s question-answering 
computer Watson’s victory in the TV game show “Jeopardy!”; and the surprising success of self-driving 
cars in the DARPA Grand Challenge competitions in the 2000s. 


The current wave of progress and enthusiasm for AI began around 2010, driven by three factors that built 
upon each other: the availability of big data from sources including e-commerce, businesses, social 
media, science, and government; which provided raw material for dramatically improved machine 
learning approaches and algorithms; which in turn relied on the capabilities of more powerful 
computers.’ During this period, the pace of improvement surprised AI experts. For example, on a popular 
image recognition challenge’ that has a 5 percent human error rate according to one error measure, the 
best AI result improved from a 26 percent error rate in 2011 to 3.5 percent in 2015. 


Simultaneously, industry has been increasing its investment in AI. In 2016, Google Chief Executive 
Officer (CEO) Sundar Pichai said, “Machine learning [a subfield of AI] is a core, transformative way by 
which we’re rethinking how we’re doing everything. We are thoughtfully applying it across all our 
products, be it search, ads, YouTube, or Play. And we’re in early days, but you will see us—in a 
systematic way—apply machine learning in all these areas.”’ This view of AI broadly impacting how 
software is created and delivered was widely shared by CEOs in the technology industry, including Ginni 
Rometty of IBM, who has said that her organization is betting the company on AI.’ 


What is Artificial Intelligence? 


There is no single definition of AI that is universally accepted by practitioners. Some define AI loosely as 
a computerized system that exhibits behavior that is commonly thought of as requiring intelligence. 
Others define AI as a system capable of rationally solving complex problems or taking appropriate 
actions to achieve its goals in whatever real world circumstances it encounters. 


Experts offer differing taxonomies of AI problems and solutions. A popular AI textbook’ used the 
following taxonomy: (1) systems that think like humans (e.g., cognitive architectures and neural 
networks); (2) systems that act like humans (e.g., pass the Turing test via natural language processing; 
knowledge representation, automated reasoning, and learning), (3) systems that think rationally (e.g., 


5 A more detailed history of AI is available in the Appendix of the AI 100 Report. Peter Stone, Rodney Brooks, Erik 
Brynjolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit 
Kraus, Kevin Leyton-Brown, David Parkes, William Press, AnnaLee Saxenian, Julie Shah, Milind Tambe, and 

Astro Teller, "Artificial Intelligence and Life in 2030," One Hundred Year Study on Artificial Intelligence: Report of 
the 2015-2016 Study Panel, Stanford University, Stanford, CA, September 2016, http://ail00.stanford.edu/2016- 
report. 


€ The ImageNet Large Scale Visual Recognition Challenge provides a set of photographic images and asks for an 
accurate description of what is depicted in each image. Statistics in the text refer to the “classification error” metric 
in the “classificationtlocalization with provided training data” task. See http://image-net.org/challenges/LSVRC/. 


7 Steven Levy, “How Google is Remaking Itself as a Machine Learning First Company,” Backchannel, June 22, 
2016, https://backchannel.com/how-google-is-remaking-itself-as-a-machine-learning-first-company-ada63defcb70. 


8 See, e.g., Andrew Nusca, “IBM’s CEO Thinks Every Digital Business Will Become a Cognitive Computing 
Business,” Fortune, June 1 2016. (“[IBM] CEO Ginni Rometty is optimistic that the company’s wager on ‘cognitive 
computing,’ the term it uses for applied artificial intelligence and machine learning technologies, is the biggest bet 
the company will make in its 105-year history.”) 


? Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach (3 Edition) (Essex, England: 
Pearson, 2009). 
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logic solvers, inference, and optimization); and (4) systems that act rationally (e.g., intelligent software 
agents and embodied robots that achieve goals via perception, planning, reasoning, learning, 
communicating, decision-making, and acting). Separately, venture capitalist Frank Chen broke down the 
problem space of AI into five general categories: logical reasoning, knowledge representation, planning 
and navigation, natural language processing, and perception.’ And AI researcher Pedro Domingos 
ascribed AI researchers to five “tribes” based on the methods they use: “symbolists” use logical reasoning 
based on abstract symbols, “connectionists” build structures inspired by the human brain; 
“evolutionaries” use methods inspired by Darwinian evolution; “Bayesians” use probabilistic inference; 
and “analogizers” extrapolate from similar cases seen previously. '! 


This diversity of AI problems and solutions, and the foundation of AI in human evaluation of the 
performance and accuracy of algorithms, makes it difficult to clearly define a bright-line distinction 
between what constitutes AI and what does not. For example, many techniques used to analyze large 
volumes of data were developed by AI researchers and are now identified as “Big Data” algorithms and 
systems. In some cases, opinion may shift, meaning that a problem is considered as requiring AI before it 
has been solved, but once a solution is well known it is considered routine data processing. Although the 
boundaries of AI can be uncertain and have tended to shift over time, what is important is that a core 
objective of AI research and applications over the years has been to automate or replicate intelligent 
behavior. 


The Current State of AI 


Remarkable progress has been made on what is known as Narrow AI, which addresses specific 
application areas such as playing strategic games, language translation, self-driving vehicles, and image 
recognition.” Narrow AI underpins many commercial services such as trip planning, shopper 
recommendation systems, and ad targeting, and is finding important applications in medical diagnosis, 
education, and scientific research. These have all had significant societal benefits and have contributed to 
the economic vitality of the Nation.! 


General AI (sometimes called Artificial General Intelligence, or AGI) refers to a notional future AI 
system that exhibits apparently intelligent behavior at least as advanced as a person across the full range 
of cognitive tasks. A broad chasm seems to separate today’s Narrow AI from the much more difficult 
challenge of General AI. Attempts to reach General AI by expanding Narrow AI solutions have made 
little headway over many decades of research. The current consensus of the private-sector expert 
community, with which the NSTC Committee on Technology concurs, is that General AI will not be 
achieved for at least decades. '* 


'0 Frank Chen, “AI, Deep Learning, and Machine Learning: A Primer,” Andreessen Horowitz, June 10, 2016, 
http://al6z.com/2016/06/10/ai-deep-learning-machines. 


11 Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our 
World (New York, New York: Basic Books, 2015). 


12 Narrow AI is not a single technical approach, but rather a set of discrete problems whose solutions rely on a 
toolkit of AI methods along with some problem-specific algorithms. The diversity of Narrow AI problems and 
solutions, and the apparent need to develop specific methods for each Narrow AI application, has made it infeasible 
to “generalize” a single Narrow AI solution to produce intelligent behavior of general applicability. 


13 Mike Purdy and Paul Daugherty, “Why Artificial Intelligence is the Future of Growth,” Accenture, 2016, 
https://www.accenture.com/us-en/_acnmedia/PDF-33/Accenture-Why-AI-is-the-Future-of-Growth.pdf. 


14 Expert opinion on the expected arrival date of AGI ranges from 2030 to centuries from now. There is a long 
history of excessive optimism about AI. For example, AI pioneer Herb Simon predicted in 1957 that computers 
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People have long speculated on the implications of computers becoming more intelligent than humans. 
Some predict that a sufficiently intelligent AI could be tasked with developing even better, more 
intelligent systems, and that these in turn could be used to create systems with yet greater intelligence, 
and so on, leading in principle to an “intelligence explosion” or “singularity” in which machines quickly 
race far ahead of humans in intelligence.!> 


In a dystopian vision of this process, these super-intelligent machines would exceed the ability of 
humanity to understand or control. If computers could exert control over many critical systems, the result 
could be havoc, with humans no longer in control of their destiny at best and extinct at worst. This 
scenario has long been the subject of science fiction stories, and recent pronouncements from some 
influential industry leaders have highlighted these fears. 


A more positive view of the future held by many researchers sees instead the development of intelligent 
systems that work well as helpers, assistants, trainers, and teammates of humans, and are designed to 
operate safely and ethically. 


The NSTC Committee on Technology’s assessment is that long-term concerns about super-intelligent 
General AI should have little impact on current policy. The policies the Federal Government should adopt 
in the near-to-medium term if these fears are justified are almost exactly the same policies the Federal 
Government should adopt if they are not justified. The best way to build capacity for addressing the 
longer-term speculative risks is to attack the less extreme risks already seen today, such as current 
security, privacy, and safety risks, while investing in research on longer-term capabilities and how their 
challenges might be managed. Additionally, as research and applications in the field continue to mature, 
practitioners of AI in government and business should approach advances with appropriate consideration 
of the long-term societal and ethical questions — in additional to just the technical questions — that such 
advances portend. Although prudence dictates some attention to the possibility that harmful super- 
intelligence might someday become possible, these concerns should not be the main driver of public 
policy for AI. 


Machine Learning 


Machine learning is one of the most important technical approaches to AI and the basis of many recent 
advances and commercial applications of AI. Modern machine learning is a statistical process that starts 
with a body of data and tries to derive a rule or procedure that explains the data or can predict future data. 
This approach—learning from data—contrasts with the older “expert system” approach to AI, in which 
programmers sit down with human domain experts to learn the rules and criteria used to make decisions, 
and translate those rules into software code. An expert system aims to emulate the principles used by 
human experts, whereas machine learning relies on statistical methods to find a decision procedure that 
works well in practice. 


An advantage of machine learning is that it can be used even in cases where it is infeasible or difficult to 
write down explicit rules to solve a problem. For example, a company that runs an online service might 
use machine learning to detect user log-in attempts that are fraudulent. The company might start with a 
large data set of past login attempts, with each attempt labeled as fraudulent or not using the benefit of 


would outplay humans at chess within a decade, an outcome that required 40 years to occur. Early predictions about 
automated language translation also proved wildly optimistic, with the technology only becoming usable (and by no 
means fully fluent) in the last several years. It is tempting but incorrect to extrapolate from the ability to solve one 
particular task to imagine machines with a much broader and deeper range of capabilities and to overlook the huge 
gap between narrow task-oriented performance and the type of general intelligence that people exhibit. 


15 Tt is far from certain that this sort of explosive growth in intelligence is likely, or even possible. Another plausible 
extrapolation from current knowledge is that machine intelligence will continue to increase gradually even after 
surpassing human intelligence. 
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hindsight. Based on this data set, the company could use machine learning to derive a rule to apply to 
future login attempts that predicts which attempts are more likely to be fraudulent and should be subjected 
to extra security measures. In a sense, machine learning is not an algorithm for solving a specific problem, 
but rather a more general approach to finding solutions for many different problems, given data about 
them. 


To apply machine learning, a practitioner starts with a historical data set, which the practitioner divides 
into a training set and a test set. The practitioner chooses a model, or mathematical structure that 
characterizes a range of possible decision-making rules with adjustable parameters. A common analogy is 
that the model is a “box” that applies a rule, and the parameters are adjustable knobs on the front of the 
box that control how the box operates. In practice, a model might have many millions of parameters. 


The practitioner also defines an objective function used to evaluate the desirability of the outcome that 
results from a particular choice of parameters. The objective function will typically contain parts that 
reward the model for closely matching the training set, as well as parts that reward the use of simpler 
rules. 


Training the model is the process of adjusting the parameters to maximize the objective function. 
Training is the difficult technical step in machine learning. A model with millions of parameters will have 
astronomically more possible outcomes than any algorithm could ever hope to try, so successful training 
algorithms have to be clever in how they explore the space of parameter settings so as to find very good 
settings with a feasible level of computational effort. 


Once a model has been trained, the practitioner can use the test set to evaluate the accuracy and 
effectiveness of the model. The goal of machine learning is to create a trained model that will 
generalize—it will be accurate not only on examples in the training set, but also on future cases that it has 
never seen before. While many of these models can achieve better-than-human performance on narrow 
tasks such as image labeling, even the best models can fail in unpredictable ways. For example, for many 
image labeling models it is possible to create images that clearly appear to be random noise to a human 
but will be falsely labeled as a specific object with high confidence by a trained model. 16 


Another challenge in using machine learning is that it is typically not possible to extract or generate a 
straightforward explanation for why a particular trained model is effective. Because trained models have a 
very large number of adjustable parameters—often hundreds of millions or more—training may yield a 
model that "works," in the sense of matching the data, but is not necessarily the simplest model that 
works. In human decision-making, any opacity in the process is typically due to not having enough 
information about why a decision was reached, because the decider may be unable to articulate why the 
decision “felt right.” With machine learning, everything about the decision procedure is known with 
mathematical precision, but there may be simply too much information to interpret clearly. 


Deep Learning 


In recent years, some of the most impressive advancements in machine learning have been in the subfield 
of deep learning, also known as deep network learning. Deep learning uses structures loosely inspired by 
the human brain, consisting of a set of units (or “neurons”’). Each unit combines a set of input values to 
produce an output value, which in turn is passed on to other neurons downstream. For example, in an 
image recognition application, a first layer of units might combine the raw data of the image to recognize 
simple patterns in the image; a second layer of units might combine the results of the first layer to 
recognize patterns-of-patterns; a third layer might combine the results of the second layer; and so on. 


16 See, e.g., Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy, “Explaining and Harnessing Adversarial 
Examples,” http://arxiv.org/pdf/1412.6572.pdf. 
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Deep learning networks typically use many layers—sometimes more than 100— and often use a large 
number of units at each layer, to enable the recognition of extremely complex, precise patterns in data. 


In recent years, new theories of how to construct and train deep networks have emerged, as have larger, 
faster computer systems, enabling the use of much larger deep learning networks. The dramatic success of 
these very large networks at many machine learning tasks has come as a surprise to some experts, and is 
the main cause of the current wave of enthusiasm for machine learning among AI researchers and 
practitioners. 


Autonomy and Automation 


Al is often applied to systems that can control physical actuators or trigger online actions. When AI 
comes into contact with the everyday world, issues of autonomy, automation, and human-machine 
teaming arise. 


Autonomy refers to the ability of a system to operate and adapt to changing circumstances with reduced or 
without human control. For example, an autonomous car could drive itself to its destination. Despite the 
focus in much of the literature on cars and aircraft, autonomy is a much broader concept that includes 
scenarios such as automated financial trading and automated content curation systems. Autonomy also 
includes systems that can diagnose and repair faults in their own operation, such as identifying and fixing 
security vulnerabilities. 


Automation occurs when a machine does work that might previously have been done by a person. The 
term relates to both physical work and mental or cognitive work that might be replaced by AI. 
Automation, and its impact on employment, have been significant social and economic phenomena since 
at least the Industrial Revolution. It is widely accepted that AI will automate some jobs, but there is more 
debate about whether this is just the next chapter in the history of automation or whether AI will affect the 
economy differently than past waves of automation have previously. 


Human-Machine Teaming 


In contrast to automation, where a machine substitutes for human work, in some cases a machine will 
complement human work. This may happen as a side-effect of AI development, or a system might be 
developed specifically with the goal of creating a human-machine team. Systems that aim to complement 
human cognitive capabilities are sometimes referred to as intelligence augmentation. 


In many applications, a human-machine team can be more effective than either one alone, using the 
strengths of one to compensate for the weaknesses of the other. One example is in chess playing, where a 
weaker computer can often beat a stronger computer player, if the weaker computer is given a human 
teammate—this is true even though top computers are much stronger players than any human.!* Another 
example is in radiology. In one recent study, given images of lymph node cells, and asked to determine 


11 Different definitions of “automation” are used in different settings. The definition used in the main text, involving 
the substitution of machine labor for human labor, is commonly used in economics. Another definition is used in the 
systems analysis setting in the Department of Defense (DoD): Automation means that the system functions with 
little or no human operator involvement. However the system performance is limited to the specific pre-programmed 
actions it has been designed to execute. Once the system is initiated by a human operator, it executes its task 
according to those instructions and subroutines, which have been tested and validated. Typically these are well- 
defined tasks that have predetermined responses, i.e., rule-based responses in reasonably well-known and structured 
environments. 

'8 Garry Kasparov, “The Chess Master and the Computer,” New York Review of Books, February 11, 2010. 
http://www.nybooks.com/articles/2010/02/1 1/the-chess-master-and-the-computer. 
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whether or not the cells contained cancer, an Al-based approach had a 7.5 percent error rate, where a 
human pathologist had a 3.5 percent error rate; a combined approach, using both AI and human input, 
lowered the error rate to 0.5 percent, representing an 85 percent reduction in error." 


1 Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun Irshad, Andrew H. Beck, “Deep Learning for 
Identifying Metastatic Breast Cancer,” June 18, 2016, https://arxiv.org/pdf/1606.05718v1.pdf. 
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Public Outreach and Development of this Report 


This report was developed by the NSTC’s Subcommittee on Machine Learning and Artificial Intelligence, 
which was chartered in May 2016 to foster interagency coordination and provide technical and policy 
advice on topics related to AI, and to monitor the development of AI technologies across industry, the 
research community, and the Federal Government. The report follows a series of public outreach 
activities led by OSTP, designed to allow government officials thinking about these topics to learn from 
experts and from the public. This public outreach on AI included five co-hosted public workshops, and a 
public Request for Information (RFI). The public workshops were: 


e AI, Law, and Governance (May 24, in Seattle, co-hosted by OSTP, the National Economic Council 
(NEC), and the University of Washington); 

e AI for Social Good (June 7, in Washington DC, co-hosted by OSTP, the Association for the 
Advancement of AI (AAAI) and the Computing Community Consortium (CCC)); 

e Future of AI: Emerging Topics and Societal Benefit at the Global Entrepreneurship Summit (June 23, 
in Palo Alto, co-hosted by OSTP and Stanford University); 

e AI Technology, Safety, and Control (June 28, in Pittsburgh, co-hosted by OSTP and Carnegie Mellon 
University); and 

e Social and Economic Impacts of AI (July 7, in New York, co-hosted by OSTP, NEC, and New York 
University). 


In conjunction with each of the five workshops, the private-sector co-hosts organized separate meetings 
or conference sessions which government staff attended. Total in-person attendance at the public events 
was more than 2,000 people, in addition to international online streaming audiences, which included more 
than 3,500 people for the Washington, DC workshop livestream alone. 


OSTP also published a Request for Information (RFI) seeking public comment on the topics of the 
workshops. The RFI closed on July 22, 2016 and received 161 responses. Comments submitted in 
response to the public RFI were published by OSTP on September 6, 2016.” 


20 Ed Felten and Terah Lyons, “Public Input and Next Steps on the Future of Artificial Intelligence,” Medium, 
September 6 2016, https://medium.com/@USCTO/public-input-and-next-steps-on-the-future-of-artificial- 
intelligence-458b82059fc3. 
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Applications of AI for Public Good 


One area of great optimism about AI and machine learning is their potential to improve people’s lives by 
helping to solve some of the world’s greatest challenges and inefficiencies. The promise of AI has been 
compared to the transformative impacts of advances in mobile computing.”! Public- and private-sector 
investments in basic and applied R&D on AI have already begun reaping major benefits for the public in 
fields as diverse as health care, transportation, the environment, criminal justice, and economic 
inclusion.” 


At Walter Reed Medical Center, the Department of Veteran Affairs is using AI to better predict medical 
complications and improve treatment of severe combat wounds, leading to better patient outcomes, faster 
healing, and lower costs.” The same general approach—predicting complications to enable preventive 
treatment—has also reduced hospital-acquired infections at Johns Hopkins University.” Given the current 
transition to electronic health records, predictive analysis of health data may play a key role across many 
health domains like precision medicine and cancer research. 


In transportation, Al-enabled smarter traffic management applications are reducing wait times, energy 
use, and emissions by as much as 25 percent in some places.” Cities are now beginning to leverage the 
type of responsive dispatching and routing used by ride-hailing services, and linking it with scheduling 
and tracking software for public transportation to provide just-in-time access to public transportation that 
can often be faster, cheaper and, in many cases, more accessible to the public. 


Some researchers are leveraging AI to improve animal migration tracking by using AI image 
classification software to analyze tourist photos from public social media sites. The software can identify 
individual animals in the photos and build a database of their migration using the data and location stamps 
on the photos. At OSTP’s AI for Social Good workshop, researchers talked about building some of the 
largest available datasets to-date on the populations and migrations of whales and large African animals, 
and about launching a project to track “The Internet of Turtles” to gain new insights about sea life.” 
Other speakers described uses of AI to optimize the patrol strategy of anti-poaching agents, and to design 
habitat preservation strategies to maximize the genetic diversity of endangered populations. 


21 Frank Chen, “AI, Deep Learning, and Machine Learning: A Primer,” Andreessen Horowitz, June 10, 2016, 
http://al6z.com/2016/06/10/a1-deep-learning-machines. 


22 The potential benefits of increasing access to digital technologies are detailed further in the World Bank Group’s 
Digital Dividends report. (“World Development Report 2016: Digital Dividends,” The World Bank Group, 2016, 
http://documents. worldbank. org/curated/en/89697 1468 19497288 1/pdf/102725-PUB-Replacement-PUBLIC.pdf.) 


3 Eric Elster, “Surgical Critical Care Initiative: Bringing Precision Medicine to the Critically Ill,” presentation at AI 
for Social Good workshop, Washington, DC, June 7, 2016, http://cra.org/ccc/wp- 
content/uploads/sites/2/20 16/06/Eric-Elster-AI-slides-min.pdf. 


24 Katharine E. Henry, David N. Hager, Peter J. Pronovost, and Suchi Saria, "A targeted real-time early warning 
score (TREWScore) for septic shock," Science Translational Medicine 7, no. 299 (2015): 299ra122-299ral1 22. 


25 Stephen F. Smith, “Smart Infrastructure for Urban Mobility,” presentation at AI for Social Good workshop, 
Washington, DC, June 7, 2016, http://cra.org/ccc/wp-content/uploads/sites/2/20 1 6/06/Stephen-Smith-AlI-slides. pdf. 


26 Aimee Leslie, Christine Hof, Diego Amorocho, Tanya Berger-Wolf, Jason Holmberg, Chuck Stewart, Stephen G. 
Dunbar, and Claire Jea,, “The Internet of Turtles,” April 12, 2016, 
https://www.researchgate.net/publication/301202821 The Internet_of Turtles. 
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Autonomous sailboats and watercraft are already patrolling the oceans carrying sophisticated sensor 
instruments, collecting data on changes in Arctic ice and sensitive ocean ecosystems in operations that 
would be too expensive or dangerous for crewed vessels. Autonomous watercraft may be much cheaper to 
operate than manned ships, and may someday be used for enhanced weather prediction, climate 
monitoring, or policing illegal fishing.’ 


Al also has the potential to improve aspects of the criminal justice system, including crime reporting, 
policing, bail, sentencing, and parole decisions. The Administration is exploring how AI can responsibly 
benefit current initiatives such as Data Driven Justice and the Police Data Initiative that seek to provide 
law enforcement and the public with data that can better inform decision-making in the criminal justice 
system, while also taking care to minimize the possibility that AI might introduce bias or inaccuracies due 
to deficiencies in the available data. 


Several U.S. academic institutions have launched initiatives to use AI to tackle economic and social 
challenges. For example, the University of Chicago created an academic program that uses data science 
and AI to address public challenges such as unemployment and school dropouts.”® The University of 
Southern California launched the Center for Artificial Intelligence in Society, an institute dedicated to 
studying how computational game theory, machine learning, automated planning and multi-agent 
reasoning techniques can help to solve socially relevant problems like homelessness. Meanwhile, 
researchers at Stanford University are using machine learning in efforts to address global poverty by 
using AI to analyze satellite images of likely poverty zones to identify where help is needed most.” 


Many uses of AI for public good rely on the availability of data that can be used to train machine learning 
models and test the performance of AI systems. Agencies and organizations with data that can be released 
without implicating personal privacy or trade secrets can help to enable the development of AI by making 
those data available to researchers. Standardizing data schemas and formats can reduce the cost and 
difficulty of making new data sets useful. 


Recommendation 1: Private and public institutions are encouraged to examine whether and how 
they can responsibly leverage AI and machine learning in ways that will benefit society. Social 
justice and public policy institutions that do not typically engage with advanced technologies and 
data science in their work should consider partnerships with AI researchers and practitioners that 
can help apply AI tactics to the broad social problems these institutions already address in other 
ways. 


Recommendation 2: Federal agencies should prioritize open training data and open data standards 
in AI. The government should emphasize the release of datasets that enable the use of AI to address 
social challenges. Potential steps may include developing an “Open Data for AI” initiative with the 
objective of releasing a significant number of government data sets to accelerate AI research and 
galvanize the use of open data standards and best practices across government, academia, and the 
private sector. 





27 John Markoff, “No Sailors Needed: Robot Sailboats Scout the Oceans for Data,” The New York Times, September 
4, 2016. 


28 “Data Science for Social Good,” University of Chicago, https://dssg.uchicago.edu/. 


2° Neal Jean, Marshall Burke, Michael Xie, W. Matthew Davis, David B. Lobell, and Stefano Ermon. "Combining 
satellite imagery and machine learning to predict poverty." Science 353, no. 6301 (2016): 790-794. 
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AI in the Federal Government 


The Administration is working to develop policies and internal practices that will maximize the economic 
and societal benefits of AI and promote innovation. These policies and practices may include: 


investing in basic and applied research and development (R&D); 

serving as an early customer for AI technologies and their applications; 

supporting pilot projects and creating testbeds in real-world settings; 

making data sets available to the public; 

sponsoring incentive prizes; 

identifying and pursuing Grand Challenges to set ambitious but achievable goals for AI; 
funding rigorous evaluations of AI applications to measure their impact and cost-effectiveness; 
and 

e creating a policy, legal, and regulatory environment that allows innovation to flourish while 
protecting the public from harm. 


Using AI in Government to Improve Services and Benefit the American People 


One challenge in using AI to improve services is that the Federal Government’s capacity to foster and 
harness innovation in order to better serve the country varies widely across agencies. Some agencies are 
more focused on innovation, particularly those agencies with large R&D budgets, a workforce that 
includes many scientists and engineers, a culture of innovation and experimentation, and strong ongoing 
collaborations with private-sector innovators. Many also have organizations that are specifically tasked 
with supporting high-risk, high-return research (e.g., the advanced research projects agencies in the 
Departments of Defense and Energy, as well as the Intelligence Community), and fund R&D across the 
full range from basic research to advanced development. Other agencies like the NSF have research and 
development as their primary mission. 


But some agencies, particularly those charged with reducing poverty and increasing economic and social 
mobility, have more modest levels of relevant capabilities, resources, and expertise.*° For example, while 
the National Institutes of Health (NIH) has an R&D budget of more than $30 billion, the Department of 
Labor’s R&D budget is only $14 million. This limits the Department of Labor’s capacity to explore 
applications of AI, such as applying Al-based “digital tutor” technology to increase the skills and incomes 
of non-college educated workers. 


DARPA’s “Education Dominance” program serves as an example of AI’s potential to fulfill and 
accelerate agency priorities. DARPA, intending to reduce from years to months the time required for new 
Navy recruits to become experts in technical skills, now sponsors the development of a digital tutor that 
uses AI to model the interaction between an expert and a novice. An evaluation of the digital tutor 
program concluded that Navy recruits using the digital tutor to become IT systems administrators 
frequently outperform Navy experts with 7-10 years of experience in both written tests of knowledge and 
real-world problem solving.*! 


Preliminary evidence based on digital tutor pilot projects also suggests that workers who have completed 
a training program that uses the digital tutor are more likely to get a high-tech job that dramatically 


30 Thomas Kalil, “A Broader Vision for Government Research,” Issues in Science and Technology, 2003. 


31 “Winning the Education Future: The Role of ARPA-ED,” The U.S. Department of Education, March 8 2011, 
https://www. whitehouse. gov/sites/default/files/microsites/ostp/arpa-ed-factsheet. pdf. 
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increases their incomes.** These wage increases appear to be much larger than the impacts of current 
workforce development programs.” Ideally, these results would be confirmed with independently 
conducted, randomized, controlled trials. Currently, the cost of developing digital tutors is high, and there 
is no repeatable methodology for developing effective digital tutors. Research that enables the emergence 
of an industry that uses AI approaches such as digital tutors could potentially help workers acquire in- 
demand skills. 


Recommendation 3: The Federal Government should explore ways to improve the capacity of key 
agencies to apply AI to their missions. For example, Federal agencies should explore the potential to 
create DARPA-like organizations to support high-risk, high-reward AI research and its application, 
much as the Department of Education has done through its proposal to create an “ARPA-ED,”’ to 
support R&D to determine whether AI and other technologies could significantly improve student 
learning outcomes. 


Recommendation 4: The NSTC MLAI subcommittee should develop a community of practice for 
AI practitioners across government. Agencies should work together to develop and share standards 
and best practices around the use of Al in government operations. Agencies should ensure that 
Federal employee training programs include relevant AI opportunities. 





32 The President’s Council of Advisors on Science and Technology, letter to the President, September 2014, 
https://www. whitehouse. gov/sites/default/files/microsites/ostp/PCAST/pcast_workforce_edit_report_sept_2014.pdf. 





33 J.D. Fletcher, “Digital Tutoring in Information Systems Technology for Veterans: Data Report,” The Institute for 
Defense Analysis, September 2014. 
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Al and Regulation 


AI has applications in many products, such as cars and aircraft, which are subject to regulation designed 
to protect the public from harm and ensure fairness in economic competition. How will the incorporation 
of AI into these products affect the relevant regulatory approaches? In general, the approach to regulation 
of Al-enabled products to protect public safety should be informed by assessment of the aspects of risk 
that the addition of AI may reduce, alongside the aspects of risk that it may increase. If a risk falls within 
the bounds of an existing regulatory regime, moreover, the policy discussion should start by considering 
whether the existing regulations already adequately address the risk, or whether they need to be adapted 
to the addition of AI. Also, where regulatory responses to the addition of AI threaten to increase the cost 
of compliance or slow the development or adoption of beneficial innovations, policymakers should 
consider how those responses could be adjusted to lower costs and barriers to innovation without 
adversely impacting safety or market fairness. 


The general consensus of the RFI commenters was that broad regulation of AI research or practice would 
be inadvisable at this time.*4 Instead, commenters said that the goals and structure of existing regulations 
were sufficient, and commenters called for existing regulation to be adapted as necessary to account for 
the effects of AI. For example, commenters suggested that motor vehicle regulation should evolve to 
account for the anticipated arrival of autonomous vehicles, and that the necessary evolution could be 
carried out within the current structure of vehicle safety regulation. In doing so, agencies must remain 
mindful of the fundamental purposes and goals of regulation to safeguard the public good, while creating 
space for innovation and growth in AI. 


Effective regulation of technologies such as AI requires agencies to have in-house technical expertise to 
help guide regulatory decision-making. The need for senior-level expert participation exists at regulating 
departments and agencies, and at all stages of the regulatory process. A range of personnel assignment 
and exchange models (e.g. hiring authorities) can be used to develop a Federal workforce with more 
diverse perspectives on the current state of technological development. One example of such an authority 
is the Intergovernmental Personnel Act (IPA) Mobility Program, which provides for the temporary 
assignment of personnel between the Federal Government and state and local governments, colleges and 
universities, Indian tribal governments, federally funded research and development centers, and other 
eligible organizations. If used strategically, the IPA program can help agencies meet their needs for hard- 
to-fill positions and increase their ability to hire candidates from diverse technical backgrounds. Federal 
employees serving in IPA assignments can serve as both recruiters and ambassadors for the Federal 
workforce. For example, agency staff sent to colleges and universities as instructors can inspire students 
to consider Federal employment. Likewise, programs that rotate employees through different jobs and 
sectors can help government employees gain knowledge and experience to inform regulation and policy, 
especially as it relates to emergent technologies like AI. 


34 Ed Felten and Terah Lyons, “Public Input and Next Steps on the Future of Artificial Intelligence.” 
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Recommendation 5: Agencies should draw on appropriate technical expertise at the senior level 
when setting regulatory policy for Al-enabled products. Effective regulation of Al-enabled products 
requires collaboration between agency leadership, staff knowledgeable about the existing regulatory 
framework and regulatory practices generally, and technical experts with knowledge of AI. Agency 
leadership should take steps to recruit the necessary technical talent, or identify it in existing agency 
staff, and should ensure that there are sufficient technical “seats at the table” in regulatory policy 


discussions. 


Recommendation 6: Agencies should use the full range of personnel assignment and exchange 
models (e.g. hiring authorities) to foster a Federal workforce with more diverse perspectives on the 
current state of technology. 





Case Study: Autonomous Vehicles and Aircraft 


A relevant example of the regulatory challenges associated with an agency updating legacy regulations to 
account for new AlJ-based products is the work of the Department of Transportation (DOT) on automated 
vehicles and unmanned aircraft systems (UAS, or “drones”’). Within DOT, automated cars are regulated 
by the National Highway Traffic Safety Administration (NHTSA) and aircraft are regulated by the 
Federal Aviation Administration (FAA). 


The Promise of Autonomy 


The application of AI to vehicles and aircraft has captured the public imagination. Today’s new cars have 
Al-based driver assist features like self-parking and advanced cruise controls that keep a car in its lane 
and adjust speed based on surrounding vehicles. Experimental fully automated cars monitored by humans 
can already be seen driving on the roads. The consensus of experts is that automated surface vehicle 
technology will eventually be safer than human drivers and may someday prevent most of the tens of 
thousands of fatalities that occur annually on the Nation’s roads and highways. 


Automated vehicles also offer the possibility of greater mobility for the elderly and Americans with 
disabilities who may not be able to drive. First- and last-mile access to transit and other novel 
transportation approaches may provide communities isolated from essential services such as jobs, health 
care, and groceries unprecedented access to opportunity. A well-designed system of automated vehicles 
able to predict and avoid collisions may also significantly reduce transportation-related emissions and 
energy consumption. The Administration is taking steps to make this vision a reality, including the 
proposed $3.9 billion investment in the President’s Fiscal Year (FY) 2017 Budget by the Department of 
Transportation in automated and connected vehicle research, development, and deployment efforts, to 
ensure that the United States maintains its lead in automated vehicle technologies.’ 


35 “Secretary Foxx Unveils President Obama’s FY17 Budget Proposal of Nearly $4 Billion for Automated Vehicles 
and Announces DOT Initiatives to Accelerate Vehicle Safety Innovations,” U.S. Department of Transportation, 
January 14 2016, https://www.transportation. gov/briefing-room/secretary-foxx-unveils-president- 
obama%E2%80%99s-fy17-budget-proposal-nearly-4-billion. 
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Moving to the air, since the early 1990s, commercial UAS have operated on a limited basis in the 
National Airspace System (NAS).*° Until recently, UAS mainly supported government operations, such 
as military and border security operations. But in recent years, potential applications have rapidly 
expanded to include aerial photography, surveying land and crops, monitoring forest fires, responding to 
disasters, and inspecting critical infrastructure. Several government agencies are already operating UAS 
to advance their missions, and thousands of Americans have obtained the necessary authority from the 
Federal Aviation Administration (FAA) for commercial UAS operations, a process accelerated under the 
FAA’s “Small UAS Rule” that took effect in August 2016 and the FAA’s Small UAS Aircraft 
Registration Service that launched in December 2015. The FAA estimates that the number of UAS 
registered for commercial use will exceed 600,000 by August 2017.°’ 


One estimate of the economic impact of integrating of UAS into the airspace predicted more than $13.6 
billion of economic value created by UAS in the first three years of integration, with sustainable growth 
predicted to follow.** A 2013 study from the Association for Unmanned Vehicle Systems International 
predicted that the commercial drone industry could generate more than $82 billion for the U.S. economy 
and create more than 100,000 new jobs over the next 10 years. Tax revenue to the states was predicted to 
increase by more than $482 million in the first decade after integration.” 


Ensuring Safety 


Realizing the potential benefits of these promising technologies requires that government take steps to 
ensure the safety of the airspace and roads, while continuing to foster a culture of innovation and growth. 
The United States has the safest and most complex aviation system in the world, and the public relies on 
FAA oversight to establish safety standards. Federal Motor Vehicle Safety Standards (FMVSS) place 
requirements on manufacturers to develop safe surface vehicles, and NHTSA has the authority to recall 
vehicles in the event of an unreasonable risk to safety. While there is considerable opportunity to reduce 
the fatalities on roads and highways, current practices result in approximately one fatality for every 100 
million vehicle miles traveled. Equaling or exceeding such performance in automated vehicles is a 
formidable challenge. 


Applying techniques of AI in such safety-critical environments raises several challenges. First among 
these is the need to translate human responsibilities while driving or flying into software. Unlike in some 
other successful applications of Narrow AI, there are no concise descriptions for the task of operating 
ground or air vehicles. Each of these operations is multifaceted, with responsibilities including guiding 
the vehicle, detecting and avoiding obstacles, and handing mechanical failures such as flat tires. While 
subtasks such as navigation or certain types of perception may align with certain existing Narrow AI 
solutions, the integration and prioritization of these tasks may not. It may seem straightforward to simply 
obey all traffic laws, but a skilled human driver may cross a double-yellow road boundary to avoid an 
accident or move past a double-parked vehicle. Though such situations may be rare, they cannot be 


3¢ The National Airspace System is the network of air navigation facilities, air traffic control facilities, airports, 
technology, and rules and regulations that are needed to protect persons and property on the ground, and to establish 
a safe and efficient airspace environment for civil, commercial, and military aviation. 


37 “Aerospace Forecast Report Fiscal Years 2016 to 2036,” The Federal Aviation Administration, March 24 016, 
https://www.faa.gov/data_research/aviation/aerospace_forecasts/media/Unmanned_Aircraft_Systems.pdf. 


38 Derryl Jenkins and Bijan Vasigh, “The Economic Impact of Unmanned Aircraft Systems Integration in the United 
States,” The Association for Unmanned Vehicle Systems International, 2013, 

https://higherlogicdownload.s3 .amazonaws.com/AUVSI/958c920a-7f9b-4ad2-9807- 
f9a4e95d1efl/UploadedImages/New_Economic%20Report%202013%20Full.pdf. 


3 Ibid. 
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ignored—simple arithmetic dictates that in order for failures to occur at least as infrequently as they do 
with human drivers, a system must handle many such rare cases without failure. 


For systems that rely on machine learning, the need to get rare cases right has implications for system 
design and testing. Machine learning approaches can be more confident that a case will be handled 
correctly if a similar case is in the training set. The challenge is how to develop a data set that includes 
enough of the rare cases that contribute to the risk of an accident. Commercial aviation has mechanisms 
for sharing incident and safety data across the industry, but reporting may not be second nature to recently 
credentialed UAS operators who are new to the safety and accountability culture of the traditional 
aviation industry. No comparable system currently exists for the automotive industry—only fatal 
accidents are reported, and the collection and reporting of other traffic safety information is done, if at all, 
in a disparate manner at the state or local level. The lack of consistently reported incident or near-miss 
data increases the number of miles or hours of operation necessary to establish system safety, presenting 
an obstacle to certain AI approaches that require extensive testing for validation. 


To facilitate safe testing, the FAA has designated six UAS Test Sites across the country and provided 
blanket authorization for UAS operations within these sites. Activities at the sites include a project to 
extend NASA’s multi-year research on UAS traffic management (UTM) to identify operational 
requirements for large-scale beyond visual line-of-sight UAS operations in low-altitude airspace. 
Similarly, ground vehicle testbeds such as the Connected Vehicle Pilots and the deployment of automated 
vehicles in Columbus, Ohio, winner of the Department of Transportation’s $40 million Smart City 
Challenge in 2016, will provide rich baseline and interaction data for AI researchers. 


Recommendation 7: The Department of Transportation should work with industry and researchers 
on ways to increase sharing of data for safety, research, and other purposes. In light of the future 
importance of AI in surface and air, Federal actors should focus in the near-term on developing 


increasingly rich sets of data, consistent with consumer privacy, that can better inform policy-making 
as these technologies mature. 





Adapting Current Regulations 


While the regulatory approaches for the Nation’s airspace and highways differ, the approaches to 
integrating autonomous vehicles and aircraft share a common goal: both the FAA and NHTSA are 
working to establish nimble and flexible frameworks that ensure safety while encouraging innovation. 


With respect to airspace regulation, a significant step toward enabling the safe integration of UAS into the 
airspace was the FAA’s promulgation of the Part 107, or “Small UAS,” final rule, which took effect on 
August 29, 2016. For the first time, the rule authorizes widespread non-recreational flights of UAS under 
55 pounds. The rule limits flights to daytime, at an altitude of 400 feet or less, with the vehicle controlled 
by a licensed operator and within the operator’s direct line of sight. Flights over people are not allowed. 
Subsequent rules are planned, to relax these restrictions as experience and data show how to do so safely. 
In particular, DOT is currently developing a Notice of Proposed Rulemaking proposing a regime for 
certain types of “micro UAS” to conduct operations over people, with a rule on expanded operations 
expected to follow. 


The FAA has not yet publicly announced a clear path to a regulation allowing fully autonomous” flight. 
Though safe integration of autonomous aircraft into the airspace will be a complex process, the FAA is 


4° This report uses the term “autonomous” for an aircraft that is controlled by a machine rather than a human. 
“Piloted” refers to an aircraft that has a human onboard who is controlling the aircraft. “Remotely-piloted” refers to 
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preparing for a not-so-distant technological future in which autonomous and piloted aircraft fly together 
in a seamlessly integrated airspace system. 


New approaches to airspace management may also include Al-based enhancement of the air traffic 
control system. Projected future air traffic densities and diversity of operations are unlikely to be feasible 
within the current airspace management architecture, due to current limits on air/ground integration, and 
reliance on human-to-human communication in air and ground practices.*! The cost of U.S. air 
transportation delays in 2007, the latest year for which there is reliable public data, was estimated to be 
$31.2 billion—a number that has presumably grown as user volume has increased since that year.” 
Though some flight delays are unavoidable due to weather and other constraints, adopting new aviation 
technologies, enabling policies, and infrastructure upgrades could significantly increase efficiency of 
operation in the U.S. airspace. Such solutions include AI and machine learning-based architectures that 
have the potential to better accommodate a wider range of airspace users, including piloted and unpiloted 
aircraft, and to use airspace more efficiently without undermining safety. Development and deployment 
of such technologies would help ensure global competitiveness for airspace users and service providers, 
while increasing safety and reducing cost.*? 


With respect to surface transportation, the most significant step currently underway to establish a 
common framework is the Federal Automated Vehicles Policy that the Administration released on 
September 20, 2016.“ The policy had several parts: 


e guidance for manufacturers, developers, and other organizations outlining a 15 point “Safety 
Assessment” for the safe design, development, testing, and deployment of highly automated 
vehicles; 

e a model state policy, which clearly distinguishes Federal and State responsibilities and 
recommends policy areas for states to consider, with a goal of generating a consistent national 
framework for the testing and operation of automated vehicles, while leaving room for 
experimentation by states; 

e an analysis of current regulatory tools that NHTSA can use to aid the safe development of 
automated vehicles, such as interpreting current rules to allow for appropriate flexibility in 
design, providing limited exemptions to allow for testing of nontraditional vehicle designs, and 
ensuring that unsafe automated vehicles are removed from the road; and 


an aircraft that is controlled by a human who is not on board. “Manned” means there is a human onboard who may 
or may not be in control. 


4! Heinz Erzberger, “The Automated Airspace Concept,” prepared for the 4th USA/Europe Air Traffic Management 
R&D Seminar Dec. 3-7, 2001, Santa Fe, New Mexico, USA, 
http://www.aviationsystemsdivision.arc.nasa.gov/publications/tactical/erzberger_12_01.pdf. 


42 Michael Ball, Cynthia Barnhart, Martin Dresner, Mark Hansen, Kevin Neels, Amedeo Odoni, Everett Peterson, 
Lance Sherry, Antonio Trani, Bo Zou, “Total Delay Impact Study: A Comprehensive Assessment of the Costs and 
Impacts of Flight Delay in the United States,” The National Center of Excellence for Aviation Operations Research, 
November 2010, http://www.nextor.org/pubs/TDI Report Final 11 _03_10.pdf. 





8 Robert W. Poole, Jr., “The Urgent Need to Reform the FAA’s Air Traffic Control System,” The Heritage 
Foundation, 2007, http://www.heritage.org/research/reports/2007/02/the-urgent-need-to-reform-the-faas-air-traffic- 
control-system. 


44 “Federal Automated Vehicles Policy,” The U.S. Department of Transportation, September 21 2016, 
https://www.transportation.gov/AV. 
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e a discussion of new tools and authorities that the agency could consider seeking in the future to 
aid the safe and efficient deployment of new lifesaving technologies and ensure that technologies 
deployed on the road are safe. 


DOT intends for the guidance and the model state policy to be routinely updated as new data are learned 
and research completed. 


Recommendation 8: The U.S. Government should invest in developing and implementing an 
advanced and automated air traffic management system that is highly scalable, and can fully 
accommodate autonomous and piloted aircraft alike. 


Recommendation 9: The Department of Transportation should continue to develop an evolving 
framework for regulation to enable the safe integration of fully automated vehicles and UAS, 
including novel vehicle designs, into the transportation system. 
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Research and Workforce 


Government also has an important role to play in advancing the AI field by investing in research and 
development, developing a workforce that is skilled and diverse, and managing the economic impacts of 
these technologies as they develop. A separate National Artificial Intelligence Research and Development 
Strategic Plan is being published in conjunction with this report. This section discusses additional policy 
issues related to research and workforce development. 


Monitoring Progress in AI 


Given the potential impacts of AI, society would benefit from accurate and timely methods for monitoring 
and forecasting AI developments.** Several projects have attempted to forecast AI futures. The 2009 
AAAI Presidential Panel on Long-Term AI Futures“ and the 2015 Future of AI Conference” brought 
together AI experts to predict the future of their field. In addition, Stanford’s One-Hundred Year Study on 
Artificial Intelligence* plans to conduct "a series of periodic studies on how AI will affect automation, 
national security, psychology, ethics, law, privacy, democracy, and other issues." The first of these studies 
was published in September 2016.” 


One potentially useful line of research is to survey expert judgments over time. As one example, a survey 
of AI researchers found that 80 percent of respondents believed that human-level General AI will 
eventually be achieved, and half believed it is at least 50 percent likely to be achieved by the year 2040. 
Most respondents also believed that General AI will eventually surpass humans in general intelligence.’ 
While these particular predictions are highly uncertain, as discussed above, such surveys of expert 
judgment are useful, especially when they are repeated frequently enough to measure changes in 
judgment over time. One way to elicit frequent judgments is to run "forecasting tournaments" such as 
prediction markets, in which participants have financial incentives to make accurate predictions.°! Other 


4 The track record of technology forecasts, in general, and AI forecasts, in particular, suggests this may be difficult. 
One of the largest retrospective reviews of technology forecasts over the last 50 years found that forecasts with time 
horizons beyond 10 years were rarely better than coin-flips**. (Carrie Mullins, “Retrospective Analysis of 
Technology Forecasting,” The Tauri Group, August 13, 2012.) One review of 95 timeline predictions for AI from 
1950 to 2012 found that most forecasts predicted General AI would be achieved "in the next 20 years." (Stuart 
Armstrong, Kaj Sotala, Sean S. OhEigeartaigh, “The errors, insights and lessons of famous AI predictions — and 
what they mean for the future,” Journal of Experimental & Theoretical Artificial Intelligence, May 20, 2014.) 


46 “A AAI Presidential Panel on Long-Term AI Futures: 2008-2009 Study,” The Association for the Advancement of 
Artificial Intelligence, http://www.aaai.org/Organization/presidential-panel.php. 


47 “AT Safety Conference in Puerto Rico,” The Future of Life Institute, October 12, 2015, 
http://futureoflife.org/2015/10/12/ai-safety-conference-in-puerto-rico. 


48 Peter Stone, et al., “Artificial Intelligence and Life in 2030,” http://ail00.stanford.edu/2016-report. 


” Ibid. 


50 Vincent Müller and Nick Bostrom, “Future progress in artificial intelligence: A Survey of Expert Opinion,” 
Fundamental Issues of Artificial Intelligence, 2014. 


5! Charles Twardy, Robin Hanson, Kathryn Laskey, Tod S. Levitt, Brandon Goldfedder, Adam Siegel, Bruce 
D’ Ambrosio, and Daniel Maxwell, “SciCast: Collective Forecasting of Innovation,” Collective Intelligence, 2014. 
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research has found that technology developments can often be accurately predicted by analyzing trends in 
publication and patent data”. 


At present, the majority of basic research in AI is conducted by academics and by commercial labs that 
regularly announce their findings and publish them in the research literature. If competition drives 
commercial labs towards increased secrecy, monitoring of progress may become more difficult, and 
public concern may increase. 


One particularly valuable line of research is to identify milestones that could represent or foreshadow 
significant leaps in AI capabilities. When asked during the outreach workshops and meetings how 
government could recognize milestones of progress in the field, especially those that indicate the arrival 
of General AI may be approaching, researchers tended to give three distinct but related types of answers: 


1. Success at broader, less structured tasks: In this view, the transition from present Narrow AI to 
an eventual General AI will occur by gradually broadening the capabilities of Narrow AI systems 
so that a single system can cover a wider range of less structured tasks. An example milestone in 
this area would be a housecleaning robot that is as capable as a person at the full range of routine 
housecleaning tasks. 


2. Unification of different “styles” of AI methods: In this view, AI currently relies on a set of 
separate methods or approaches, each useful for different types of applications. The path to 
General AI would involve a progressive unification of these methods. A milestone would involve 
finding a single method that is able to address a larger domain of applications that previously 
required multiple methods. 


3. Solving specific technical challenges, such as transfer learning: In this view, the path to General 
AI does not lie in progressive broadening of scope, nor in unification of existing methods, but in 
progress on specific technical grand challenges, opening up new ways forward. The most 
commonly cited challenge is transfer learning, which has the goal of creating a machine learning 
algorithm whose result can be broadly applied (or transferred) to a range of new applications. For 
example, transfer learning might allow a model to be trained to translate English to Spanish, in 
such a way that the resulting model could “transfer” its knowledge to similar tasks such as 
Chinese to French translation, or writing poetry in Russian, enabling these new tasks to be 
learned much more quickly. 


Recommendation 10: The NSTC Subcommittee on Machine Learning and Artificial Intelligence 
should monitor developments in AI, and report regularly to senior Administration leadership about 
the status of AI, especially with regard to milestones. The Subcommittee should update the list of 
milestones as knowledge advances and the consensus of experts changes over time. The 
Subcommittee should consider reporting to the public on AI developments, when appropriate. 


Recommendation 11: The Government should monitor the state of AI in other countries, especially 
with respect to milestones. 


Recommendation 12: Industry should work with government to keep government updated on the 
general progress of AI in industry, including the likelihood of milestones being reached soon. 





52 Sara Reardon, “Text-mining offers clues to success: US intelligence programme analyses language in patents and 
papers to identify next big technologies,” Nature no. 509, 410 (May 22 2014). 
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Federal Support for AI Research 


In 2015, the U.S. Government’s investment in unclassified R&D in Al-related technologies was 
approximately $1.1 billion, with preliminary estimates showing growth to $1.2 billion in 2016. 
Throughout the workshops and public outreach on AI conducted by OSTP, government officials heard 
calls for greater government investment in AI research and development, from business leaders, 
technologists, and economists. 


Leading researchers in AI were optimistic about sustaining the recent rapid progress in AI and its 
application to an ever wider range of applications. At the same time they emphasized that there are many 
deep unanswered questions, and no clear path toward General AI. 


Researchers reported that enthusiasm for and investment in AI research has fluctuated over recent 
decades—one low period was known as the “AI winter”—and they emphasized the importance of 
sustained investment given the history of major computer science advances taking 15 years or more to 
transition from conception in the lab to industrial maturity. 


A strong case can be made in favor of increased Federal funding for research in AI. Analysis by the 
Council of Economic Advisers (CEA) indicates that beyond AI, across all research areas, doubling or 
tripling research investment would be a net positive for the Nation due to the resulting increase in 
economic growth.® Although it may not be feasible fiscally to increase funding for all research by that 
amount, a targeted increase in areas of high economic and strategic value may offer many benefits with 
much smaller budgetary impact than an across-the-board increase. AI qualifies as a high-leverage area, 
and research agencies report that the AI research community can absorb a significant funding increase 
productively, leading to faster progress on AI and a larger cadre of trained AI practitioners. In a speech 
delivered at an AI workshop in New York City in July 2016, CEA Chairman Jason Furman said, “We 
have had substantial innovation in robotics, AI, and other areas in the last decade. But we will need a 
much faster pace of innovation in these areas to really move the dial on productivity growth going 
forward,” noting that the biggest worry that he had about AI is “that we do not have enough of [it].”%4 
To be sure, the private sector will be the main engine of progress on AI. But as it stands, there is an 
underinvestment in basic research—tresearch with long time horizons conducted for the sole purpose of 
furthering the scientific knowledge base—in part because it is difficult for a private firm to get a return 
from its investment in such research in a reasonable time frame. Basic research benefits everyone, but 
only the firm doing the research pays the costs. The literature suggests that, as a result, current levels of 
R&D spending are half to one-quarter of the level of R&D investment that would produce the optimal 
level of economic growth.°> 


5 Jason Furman, “Is This Time Different? The Opportunities and Challenges of Artificial Intelligence,” 
(presentation, AI Now: The Social and Economic Implications of Artificial Intelligence Technologies in the Near 
Term, New York, NY, July 7, 2016), Available at 

https://www. whitehouse. gov/sites/default/files/page/files/20160707_cea_ai_furman.pdf. 


54 Jason Furman, “Is This Time Different? The Opportunities and Challenges of Artificial Intelligence.” 
55 Nicholas Bloom, Mark Schankerman, John Van Reene, “Identifying Technology Spillovers and Product Market 
Rivalry,” Econometrica, 81: 1347—1393. doi:10.3982/ECTA9466. 
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Recommendation 13: The Federal government should prioritize basic and long-term AI research. 
The Nation as a whole would benefit from a steady increase in Federal and private-sector AI R&D, 


with a particular emphasis on basic research and long-term, high-risk research initiatives. Because 
basic and long-term research especially are areas where the private sector is not likely to invest, 
Federal investments will be important for R&D in these areas. 





Workforce Development and Diversity 


The rapid growth of AI has dramatically increased the need for people with relevant skills to support and 
advance the field. The AI workforce includes AI researchers who drive fundamental advances in AI, a 
larger number of specialists who refine AI methods for specific applications, and a much larger number 
of users who operate those applications in specific settings. For researchers, AI training is inherently 
interdisciplinary, often requiring a strong background in computer science, statistics, mathematical logic, 
and information theory.*° For specialists, training typically requires a background in software engineering 
and in the application area. For users, familiarity with AI technologies is needed to apply AI technologies 
reliably. 


The Role of Government 


The AI workforce challenge is in part a science, technology, engineering, and mathematics (STEM) 
education challenge that remains a priority focus of the NSTC, OSTP, and other agencies. NSF and the 
Department of Education are working with the private sector and across government to advance education 
quality, flexibility, and domain impact, to address goals such as sustained economic development, 
increased inclusion and diversity, and improved outcome measures. The NSTC Committee on Science, 
Technology and Mathematics Education (CoSTEM) brings together Federal agencies supporting STEM 
education programs to coordinate efforts on multiple topics, including AI education. 


AI knowledge and education are increasingly emphasized in Federal STEM education programs. There 
are several key roles for the Federal government in AI workforce development, including supporting 
graduate students, funding research on AI curriculum design and impact, and accrediting AI education 
programs. 


The Role of Schools and Universities 


Integrating AI, data science, and related fields throughout the Nation’s education system is essential to 
developing a workforce that can address national priorities. Educational institutions are establishing and 
growing AI programs at all levels. Universities, colleges, and even secondary schools are expanding AI 
and data science curricula, but more programs and teachers are needed. 


There are several key roles for academic institutions: 


Bronwyn H. Hall, Jacques Mairesse, and Pierre Mohnen, “Measuring the Returns to R&D,” Chapter prepared for the 
Handbook of the Economics of Innovation, B. H. Hall and N. Rosenberg (editors), December 10, 2009, 
https://eml.berkeley.edu/~bhhall/papers/HallMairesseMohnen09_rndsurvey_HEI.pdf. 


Charles I. Jones and John C. Williams, “Measuring the Social Returns to R&D,” The Quarterly Journal of 
Economics (1998) 113 (4): 1119-1135, doi: 10.1162/003355398555856. 


56 There is also a need for the development of a strong research community in fields outside of technical disciplines 
related to AI, to examine the impacts and implications of AI on economics, social science, health, and other areas of 
research. 
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e building and sustaining the researcher workforce, including computer scientists, statisticians, 
database and software programmers, curators, librarians, and archivists with specialization in data 
science; 

e training the specialist workforce, by emphasizing AI methods within software development 
courses, offering applied AI courses that demonstrate the applications of AI to other domains, and 
incorporating AI and data science challenges posed by industry, civil society, and government 
into active case studies; 

e ensuring that the user workforce has the necessary familiarity with AI systems to meet the needs 
of users and of institutions across industry, government, and academia; 

e supporting training through seed grants, professional development stipends, internships, 
fellowships, and summer research experiences; and 

e recruiting and retaining faculty, as industrial salaries grow faster than academic salaries for 
skilled researchers. 


Community colleges, two-year colleges, and certificate programs play an important role in providing 
opportunities for students and professionals to acquire necessary skills for a modest investment of their 
time and money. These opportunities may be especially relevant to workers expanding their skills, 
veterans returning to the workforce, and unemployed people seeking a way to reenter the workforce. 


An Al-enabled world demands a data-literate citizenry that is able to read, use, interpret, and 
communicate about data, and participate in policy debates about matters affected by AI. Data science 
education as early as primary or secondary school can help to improve nationwide data literacy, while 
also preparing students for more advanced data science concepts and coursework after high school. 


AI education is also a component of Computer Science for All, the President’s initiative to empower all 
American students from kindergarten through high school to learn computer science and be equipped with 
the computational thinking skills they need to be creators, not just consumers, in the digital economy, and 
to be active citizens in a technology-driven world. The American economy is rapidly shifting, and both 
educators and business leaders are increasingly recognizing that computer science (CS) is a “new basic” 
skill necessary for economic opportunity and social mobility. CS for All builds on efforts already being 
led by parents, teachers, school districts, states, and private sector leaders from across the country and is 
one way to meet the challenge of preparing a future workforce for the needs of an AI-driven economy. 


The Diversity Challenge 


All sectors face the challenge of how to diversify the AI workforce. The lack of gender and racial 
diversity in the AI workforce mirrors the lack of diversity in the technology industry and the field of 
computer science generally. Unlocking the full potential of the American people, especially in STEM 
fields, in entrepreneurship, and in the technology industry is a priority of this Administration. The 
importance of including individuals from diverse backgrounds, experiences, and identities, especially 
women and members of racial and ethnic groups traditionally underrepresented in STEM, is one of the 
most critical and high-priority challenges for computer science and AI. 


Just 18 percent of computer science graduates today are women, down from a peak of 37 percent in 
1984.°7 Though there is a lack of consistently-reported demographic data on the AI workforce, some 
statistics are available. At the Neural Information Processing Systems (NIPS) Conference in 2015—one 
of the year’s largest conferences on AI research—just 13.7 percent of conference participants were 


57 Christianne Corbett and Catherine Hill, “Solving the Equation: The Variables for Women’s Success in 
Engineering and Computing,” The American Association of University Women, March 2015, 
http://www.aauw.org/files/2015/03/Solving-the-Equation-report-nsa.pdf. 
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female." After seeing similarly low representation at a machine intelligence conference, at which she was 
the only female speaker from industry, the CEO and Co-Founder of Textio, a startup that applies AI to the 
text of job postings and recruiting emails, decided to further investigate recruitment language in the 
industry. When the company analyzed 78,768 engineering job listings, they found that job postings for 
software engineers in the machine intelligence sector had a gender-bias score in favor of men more than 
twice as high as any other sector.” 


The diversity challenge is not limited to gender. African Americans, Hispanics, and members of other 
racial and ethnic minority groups are severely underrepresented, compared to their shares of the U.S. 
population, in the STEM workforce, in computer science, and in the technology industry workforce, 
including in the field of AI. 


Many of the comments submitted to the OSTP RFI discussed the diversity challenge. Commenters 
focused on the importance of AI being produced by and for diverse populations. Doing so helps to avoid 
the negative consequences of narrowly focused AI development, including the risk of biases in 
developing algorithms, by taking advantage of a broader spectrum of experience, backgrounds, and 
opinions. These topics were also covered extensively during the public workshops. There is some 
research on the effects of a lack of diversity in the AI workforce on AI technology design and on the 
societal impacts of AI. This rich body of research is growing but still lagging behind the literature on 
broader AI workforce development needs. More research would be beneficial. 


Recommendation 14: The NSTC Subcommittees on MLAI and NITRD, in conjunction with the 
NSTC Committee on Science, Technology, Engineering, and Education (CoSTEM), should initiate 


a study on the AI workforce pipeline in order to develop actions that ensure an appropriate 
increase in the size, quality, and diversity of the workforce, including AI researchers, specialists, 
and users. 





58 Jack Clark, “Artificial Intelligence Has a ‘Sea of Dudes’ Problem,” Bloomberg, June 21, 2016, 
https://www.bloomberg.com/news/articles/20 1 6-06-23/artificial-intelligence-has-a-sea-of-dudes-problem. 


®© The next three high-scoring sectors were back-end engineering, full-stack engineering, and general software 
engineering. See more: https://textio.ai/gendered-language-in-your-job-post-predicts-the-gender-of-the-person- 
youll-hire-cd150452407d#.rht0s 1 60v. 
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Al, Automation, and the Economy 


Al’s central economic effect in the short term will be the automation of tasks that could not be automated 
before. There is some historical precedent for waves of new automation from which we can learn, and 
some ways in which AI will be different. Government must understand the potential impacts so it can put 
in place policies and institutions that will support the benefits of AI, while mitigating the costs. 


Like past waves of innovation, AI will create both benefits and costs. The primary benefit of previous 
waves of automation has been productivity growth; today’s wave of automation is no different. For 
example, a 2015 study of robots in 17 countries found that they added an estimated 0.4 percentage point 
on average to those countries’ annual GDP growth between 1993 and 2007, accounting for just over one- 
tenth of those countries’ overall GDP growth during that time.*! 


One important concern arising from prior waves of automation, however, is the potential impact on 
certain types of jobs and sectors, and the resulting impacts on income inequality. Because AI has the 
potential to eliminate or drive down wages of some jobs, especially low- and medium-skill jobs, policy 
interventions will likely be needed to ensure that AI’s economic benefits are broadly shared and that 
inequality is diminished and not worsened as a consequence. 


The economic policy questions raised by AI-driven automation are important but they are best addressed 
by a separate White House working group. The White House will conduct an additional interagency study 
on the economic impact of automation on the economy and recommended policy responses, to be 
published in the coming months. 


Recommendation 15: The Executive Office of the President should publish a follow-on report by 


the end of this year, to further investigate the effects of AI and automation on the U.S. job market, 
and outline recommended policy responses. 





6° Jason Furman, “Is This Time Different? The Opportunities and Challenges of Artificial Intelligence.” 


6! Georg Graetz and Guy Michaels, “Robots at Work,” CEPR Discussion Paper No. DP10477, March 2015, 
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2575781. 
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Fairness, Safety, and Governance 


As AI technologies gain broader deployment, technical experts and policy analysts have raised concerns 
about unintended consequences. The use of AI to make consequential decisions about people, often 
replacing decisions made by human actors and institutions, leads to concerns about how to ensure justice, 
fairness, and accountability—the same concerns voiced previously in the “Big Data” context.© The use of 
AI to control physical-world equipment leads to concerns about safety, especially as systems are exposed 
to the full complexity of the human environment. 


At a technical level, the challenges of fairness and safety are related. In both cases, practitioners strive to 
prevent intentional discrimination or failure, to avoid unintended consequences, and to generate the 
evidence needed to give stakeholders justified confidence that unintended failures are unlikely. 


Justice, Fairness, and Accountability 


A common theme in the Law and Governance, AI for Social Good, and Social and Economic Impacts 
workshops was the need to ensure that AI promotes justice and fairness, and that Al-based processes are 
accountable to stakeholders. This issue was highlighted previously in the Administration’s first Big Data 
report® published in May 2014, and the follow-up report on Big Data, Algorithmic Systems, Opportunity, 
and Civil Rights,“ published in May 2016. 


In the criminal justice system, some of the biggest concerns with Big Data are the lack of data and the 
lack of quality data. AI needs good data. If the data is incomplete or biased, AI can exacerbate problems 
of bias. It is important that anyone using AI in the criminal justice context is aware of the limitations of 
current data. 


A commonly cited example at the workshops is the use of apparently biased “risk prediction” tools by 
some judges in criminal sentencing and bail hearings as well as by some prison officials in assignment 
and parole decisions, as detailed in an extensively researched ProPublica article.® The article presented 
evidence suggesting that a commercial risk scoring tool used by some judges generates racially biased 
risk scores. A separate report from Upturn questioned the fairness and efficacy of some predictive 
policing tools.” 


® The White House, “Big Data: Seizing Opportunities, Preserving Values,” May 2014, 

https://www. whitehouse. gov/sites/default/files/docs/big_data_privacy_report_may_ 1 2014.pdf; and The White 
House, “Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights,” May 2016, 

https://www. whitehouse. gov/sites/default/files/microsites/ostp/2016 0504 data_discrimination.pdf. 





63 The White House, “Big Data: Seizing Opportunities, Preserving Values,” Executive Office of the President, May 
2014. 


4 The White House, “Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights,” Executive Office 
of the President, May 2016. 


65 Matt Ford, “The Missing Statistics of Criminal Justice,” The Atlantic, May 31, 2015, 
http://www.theatlantic.com/politics/archive/2015/05/what-we-dont-know-about-mass-incarceration/394520/ 


6 Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner, “Machine Bias,” ProPublica, May 23, 2016, 
https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. 


67 David Robinson and Logan Koepke, “Stuck in a Pattern: Early evidence on ‘predictive policing’ and civil rights,” 
Upturn, August 2016, http://www.stuckinapattern.org. 
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Similar issues could impact hiring practices. If a machine learning model is used to screen job applicants, 
and if the data used to train the model reflects past decisions that are biased, the result could be to 
perpetuate past bias. For example, looking for candidates who resemble past hires may bias a system 
toward hiring more people like those already on a team, rather than considering the best candidates across 
the full diversity of potential applicants. 


In response to these concerns, several workshop speakers argued for greater transparency when AI tools 
are used for public purposes. One speaker compared the role of AI to the role of administrative agencies 
in public decision-making. Authority is delegated to an agency due to the agency’s subject-matter 
expertise, but the delegation is constrained by due process protections, measures promoting transparency 
and oversight, and limits on the scope of the delegated authority. Some speakers called for the 
development of an analogous theory of how to maintain accountability when delegating decision-making 
power to machines. Transparency concerns focused not only on the data and algorithms used, but also on 
the potential to have some form of explanation for any AI-based determination. 


At the same workshops, AI experts cautioned that there are inherent challenges in trying to understand, 
predict, and explain the behavior of advanced AI systems, due to the complexity of the systems and the 
large volume of data they use. 


The difficulty of understanding machine learning results is at odds with the common misconception that 
complex algorithms always do what their designers choose to have them do, and therefore that bias will 
creep into an algorithm if and only if its developers themselves suffer from conscious or unconscious 
bias. It is certainly true that a technology developer who wants to produce a biased algorithm can do so, 
and that unconscious bias may cause practitioners to apply insufficient effort to preventing bias. In 
practice, however, unbiased developers with the best intentions can inadvertently produce systems with 
biased results, because even the developers of an AI system may not understand it well enough to prevent 
unintended outcomes. 


Moritz Hardt suggested an illustrative example of how bias might emerge unintentionally from the 
machine learning process.™ He postulated a machine learning model trained to distinguish people’s real 
names from false names. The model might determine that a name is more likely to be false if the first- 
name part of it is unique in the data set. This rule might have predictive power across the whole 
population, because false names are more likely to be fanciful and therefore unique. However, if there is 
an ethnic group that is a small minority of the population and tends to use a different set of first names 
than the majority population, these distinctive names are more likely to be unique in the sample, and 
therefore more likely to be incorrectly classified as false names. This effect would arise not because of 
any special treatment of the minority group’s names, and not because the input data is unrepresentative of 
the overall population, but simply because the minority group is less numerous.” 


Andrew Moore, the Dean of Computer Science at Carnegie Mellon University, offered a perspective on 
the challenge of AI and unforeseen consequences at the workshop on AI Technology, Safety, and Control. 


68 Moritz Hardt, “How big data is unfair,” Medium, September 26 2014, https://medium.com/@mrtz/how-big-data- 
is-unfair-9aa544d739de. 


© Some online services require that users sign up for accounts using their real names. Some such services use AI 
models to detect names suspected of being false, in order to cancel the associated accounts. In such a system, a user 
whose name is incorrectly classified as false may be unable to sign up for an account, or may have their account 
canceled unexpectedly. 


7° Hardt points to another way that disparate impact may occur. ML models typically become more accurate as the 
number of examples in the training set increases. In some circumstances, this may cause prediction to be more 
accurate for a majority group than for a minority. Again, this disparity arises simply because the majority group is 
more numerous, even if the dataset is representative of the population. 
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He argued that today, because of the opacity of AI algorithms, the most effective way to minimize the risk 
of unintended outcomes is through extensive testing—essentially to make a long list of the types of bad 
outcomes that could occur, and to rule out these outcomes by creating many specialized tests to look for 
them. 


An example of what can go wrong in the absence of extensive testing comes from a trained model for 
automatically captioning photos, which infamously put the caption “gorilla” on some close-up photos of 
dark-skinned human faces. This was antithetical to the developers’ values, and it occurred despite testing 
that showed the model produced accurate results on a high percentage of all photos. These particular 
errors, although rare, had negative consequences that were beyond the understanding of the model, which 
had no built-in concept of race, nor any understanding of the relevant historical context. One way to 
prevent this type of error would have involved extensive testing of the algorithm to scrutinize how human 
faces, in particular, are labeled, including examination of some results by people who could recognize 
unacceptable outcomes that the model wouldn’t catch. 


Ethical training for AI practitioners and students is a necessary part of the solution. Ideally, every student 
learning AI, computer science, or data science would be exposed to curriculum and discussion on related 
ethics and security topics.’' However, ethics alone is not sufficient. Ethics can help practitioners 
understand their responsibilities to all stakeholders, but ethical training needs to be augmented with the 
technical capability to put good intentions into practice by taking technical precautions as a system is built 
and tested. 


As practitioners strive to make AI systems more just, fair and accountable, there are often opportunities to 
make technology an aid to accountability rather than a barrier to it. Research to improve the 
interpretability of machine learning results is one example. Having an interpretable model that helps 
people understand a decision empowers them to interrogate the assumptions and processes behind it. 


There are several technical approaches to enhancing the accountability and robustness of complex 
algorithmic decisions. A system can be tested “in the wild” by presenting it with situations and observing 
its behavior. A system can be subjected to black-box testing, in which it is presented with synthetic inputs 
and its behavior is observed, enabling behavior to be tested in scenarios that might not occur naturally.” 
Some or all of the technical details of a system’s design can be published, enabling analysts to replicate it 
and analyze aspects of its internal behavior that might be difficult to characterize with testing alone. In 
some cases it is possible to publish information that helps the public evaluate a system’s risk of bias, 
while withholding other information about the system as proprietary or private. 


Safety and Control 


At the workshops, AI experts said that one of the main factors limiting the deployment of AI in the real 
world is concern about safety and control. If practitioners cannot achieve justified confidence that a 
system is safe and controllable, so that deploying the system does not create an unacceptable risk of 
serious negative consequences, then the system cannot and should not be deployed. 


™ Some institutions may choose to incorporate ethics into existing courses. Others may choose to introduce separate 
courses on ethics. 


72 Black-box testing allows a system to be presented with fictionalized data, which enables comprehensive 
experiments that vary individual attributes of an individual as well as larger numbers of experiments than might be 
possible for in-the-wild testing. See, e.g., Anupam Datta, Shayak Sen, and Yair Zick, “Algorithmic Transparency via 
Quantitative Input Influence: Theory and Experiments with Learning Systems,” Proceedings of 37" IEEE 
Symposium on Security and Privacy, 2016. 
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A major challenge in safety and control is building systems that can safely transition from the “closed 
world” of the laboratory into the outside “open world” where unpredictable things can happen. In the 
open world, a system is likely to encounter objects and situations that were not anticipated when it was 
designed and built. Adapting gracefully to unforeseen situations is difficult yet necessary for safe 
operation. 


On the topic of safety and predictability in AI, several speakers referenced a recent paper entitled 
“Concrete Problems in AI Safety,”” and the first author of the paper spoke at the workshop on 
Technology, Safety, and Control. The paper uses a running example of an autonomous robot that does 
housecleaning. The paper’s overview section gives an extended list of the sorts of practical problems that 
arise in making such a robot effective and safe, which is quoted here: 





Avoiding Negative Side Effects: How can we ensure that our cleaning robot will not disturb the 
environment in negative ways while pursuing its goals, e.g., by knocking over a vase because it 
can clean faster by doing so? Can we do this without manually specifying everything the robot 
should not disturb? 


Avoiding Reward Hacking: How can we ensure that the cleaning robot won’t game its reward 
function? For example, if we reward the robot for achieving an environment free of messes, it 
might disable its vision so that it won’t find any messes, or cover over messes with materials it 
can’t see through, or simply hide when humans are around so they can’t tell it about new types of 
messes. 





Scalable Oversight: How can we efficiently ensure that the cleaning robot respects aspects of the 
objective that are too expensive to be frequently evaluated during training? For instance, it should 
throw out things that are unlikely to belong to anyone, but put aside things that might belong to 
someone (it should handle stray candy wrappers differently from stray cellphones). Asking the 
humans involved whether they lost anything can serve as a check on this, but this check might 
have to be relatively infrequent—can the robot find a way to do the right thing despite limited 
information? 





Safe Exploration: How do we ensure that the cleaning robot doesn’t make exploratory moves 
with very bad repercussions? For example, the robot should experiment with mopping strategies, 
but putting a wet mop in an electrical outlet is a very bad idea. 





Robustness to Distributional Shift: How do we ensure that the cleaning robot recognizes, and 
behaves robustly, when in an environment different from its training environment? For example, 
heuristics it learned for cleaning factory work floors may be outright dangerous in an office. 





These examples illustrate how the “intelligence” of an AI system can be deep but narrow: the system 
might have a superhuman ability to detect dirt and optimize its mopping strategy, yet not know to avoid 
swiping a wet mop over an electrical outlet. One way to describe this overall problem is: how can we give 
intelligent machines common sense? Researchers are making slow progress on these sorts of problems. 


AI Safety Engineering 


A common theme at the Technology, Safety, and Control workshop was the need to connect open-world 
AI methods with the broader field of safety engineering. Experience in building other types of safety- 
critical systems, such as aircraft, power plants, bridges, and vehicles, has much to teach AI practitioners 
about verification and validation, how to build a safety case for a technology, how to manage risk, and 
how to communicate with stakeholders about risk. 


® Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, “Concrete 
Problems in AI Safety,” https://arxiv.org/abs/1606.06565. 
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At present, the practice of AI, especially in fast-moving areas of machine learning, can be as much art as 
science. Certain aspects of practice are not backed by a well-developed theory but instead rely on intuitive 
judgment and experimentation by practitioners. This is not unusual in newly emerging areas of 
technology, but it does limit the application of the technology in practice. Some stakeholders have 
suggested a need to grow AI into a more mature engineering field. 


As engineering fields mature, they typically move from an initial “craft” stage characterized by intuition- 
driven creation by talented amateurs and a do-it-yourself spirit; to a second commercial stage involving 
skilled practitioners, pragmatic improvement, widely accepted rules-of-thumb, and organized 
manufacture for sale; to a mature stage that integrates more rigorous methods, educated professionals, 
well-established theory, and greater specialization of products.” Most engineering fields, having a much 
longer history than modern AI, have reached a mature stage. 


In general, mature engineering fields have greater success in creating systems that are predictable, 
reliable, robust, safe, and secure. Continuing the progress toward AI becoming a mature engineering field 
will be one of the key enablers of safety and controllability as more complex systems are built. 


Recommendation 16: Federal agencies that use Al-based systems to make or provide decision 
support for consequential decisions about individuals should take extra care to ensure the efficacy 
and fairness of those systems, based on evidence-based verification and validation. 


Recommendation 17: Federal agencies that make grants to state and local governments in support 
of the use of Al-based systems to make consequential decisions about individuals should review the 
terms of grants to ensure that Al-based products or services purchased with Federal grant funds 
produce results in a sufficiently transparent fashion and are supported by evidence of efficacy and 
fairness. 


Recommendation 18: Schools and universities should include ethics, and related topics in security, 
privacy, and safety, as an integral part of curricula on AI, machine learning, computer science, 
and data science. 


Recommendation 19: AI professionals, safety professionals, and their professional societies should 
work together to continue progress toward a mature field of AI safety engineering. 





74 See, e.g., Mary Shaw, Prospects for an Engineering Discipline of Software, IEEE Software 7(6), November 1990. 
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Global Considerations and Security 


In addition to the long-term challenges of AI and the specific issues relating to fairness and safety, AI 
poses consequential policy questions in international relations, cybersecurity, and defense. 


International Cooperation 


AI has been a topic of interest in recent international discussions as countries, multilateral institutions, 
and other stakeholders have begun to assess the benefits and challenges of AI. Dialogue and cooperation 
between these entities could help advance AI R&D and harness AI for good, while also addressing 
pertinent challenges. In particular, several breakthroughs in AI are the direct or indirect result of 
collaborative research involving people, resources, and institutions in multiple countries. As with other 
digital policies, countries will need to work together to identify opportunities for cooperation and develop 
international frameworks that will help promote AI R&D and address any challenges. The United States, 
a leader in AI R&D, can continue to play a key role in global research coordination through government- 
to-government dialogues and partnerships. 


International engagement is necessary to fully explore the applications of AI in health care, automation in 
manufacturing, and information and communication technologies (ICTs). AI applications also have the 
potential to address global issues such as disaster preparedness and response, climate change, wildlife 
trafficking, the digital divide, jobs, and smart cities. The State Department foresees privacy concerns, 
safety of autonomous vehicles, and AI’s impact on long-term employment trends as AI-related policy 
areas to watch in the international context. 


In support of U.S. foreign policy priorities in this space—including ensuring U.S. international leadership 
and economic competitiveness—the U.S. Government has engaged on AI R&D and policy issues in 
bilateral discussions with other countries, including Japan, the Republic of Korea, Germany, Poland, the 
United Kingdom, and Italy, as well as in multilateral fora. International AI policy issues and the economic 
impacts of AI have also been raised in the UN, the G-7, the Organization for Economic Cooperation and 
Development (OECD), and the Asia-Pacific Economic Cooperation (APEC). The U.S. Government 
expects AI to be a topic of increasing interest in international engagements. 


The United States has been committed to working with industry and relevant standards organizations, in 
order to facilitate the development of international standards in a manner that is industry-led; voluntary; 
consensus-driven; and based on principles of transparency, openness, and market needs. The U.S. 
approach is formalized in law (NTTAA, PL 104-113) and policy (OMB Circular A-119) and reiterated in 
the United States Standards Strategy.” 


Recommendation 20: The U.S. Government should develop a government-wide strategy on 
international engagement related to AI, and develop a list of AI topical areas that need 
international engagement and monitoring. 


Recommendation 21: The U.S. Government should deepen its engagement with key international 
stakeholders, including foreign governments, international organizations, industry, academia, and 
others, to exchange information and facilitate collaboration on AI R&D. 





3 United States Standards Strategy Committee, "United States standards strategy," New York: American National 
Standards Institute (2015), 
https://share.ansi.org/shared%20documents/Standards%20Activities/NSSC/USSS_Third_edition/ANSI_USSS_2015 
pdf. 
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AI and Cybersecurity 


Today’s Narrow AI has important applications in cybersecurity, and is expected to play an increasing role 
for both defensive (reactive) measures and offensive (proactive) measures. 


Currently, designing and operating secure systems requires a large investment of time and attention from 
experts. Automating this expert work, partially or entirely, may enable strong security across a much 
broader range of systems and applications at dramatically lower cost, and may increase the agility of 
cyber defenses. Using AI may help maintain the rapid response required to detect and react to the 
landscape of ever evolving cyber threats. There are many opportunities for AI and specifically machine 
learning systems to help cope with the sheer complexity of cyberspace and support effective human 
decision making in response to cyberattacks. 


Future AI systems could perform predictive analytics to anticipate cyberattacks by generating dynamic 
threat models from available data sources that are voluminous, ever-changing, and often incomplete. 
These data include the topology and state of network nodes, links, equipment, architecture, protocols, and 
networks. AI may be the most effective approach to interpreting these data, proactively identifying 
vulnerabilities, and taking action to prevent or mitigate future attacks. 


Results to-date in DARPA’s Cyber Grand Challenge (CGC) competition demonstrate the potential of this 
approach.’° The CGC was designed to accelerate the development of advanced, autonomous systems that 
can detect, evaluate, and patch software vulnerabilities before adversaries have a chance to exploit them. 
The CGC Final Event was held on August 4, 2016. To fuel follow-on research and parallel competition, 
all of the code produced by the automated systems during the CGC Final Event has been released as open 
source to allow others to reverse engineer it and learn from it. 


AI systems also have their own cybersecurity needs. AI-driven applications should implement sound 
cybersecurity controls to ensure integrity of data and functionality, protect privacy and confidentiality, 
and maintain availability. The recent Federal Cybersecurity R&D Strategic Plan” highlighted the need for 
“sustainably secure systems development and operation.” Advances in cybersecurity will be critical in 
making AI solutions secure and resilient against malicious cyber activities, particularly as the volume and 
type of tasks conducted by governments and private sector businesses using Narrow AI increases. 


Finally, AI could support planning, coordinating, integrating, synchronizing, and directing activities to 
operate and defend U.S. government networks and systems effectively, provide assistance in support of 
secure operation of private-sector networks and systems, and enable action in accordance with all 
applicable laws, regulations and treaties. 


76 https://www.cybergrandchallenge.com 


77 “Federal Cybersecurity Research and Development Strategic Plan,” Executive Office of the President, February 
2016, 

https://www. whitehouse. gov/sites/whitehouse. gov/files/documents/2016_Federal_Cybersecurity Research _and_ De 
velopment_Stratgeic_Plan.pdf. 
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Recommendation 22: Agencies’ plans and strategies should account for the influence of AI on 
cybersecurity, and of cybersecurity on AI. Agencies involved in AI issues should engage their U.S. 
Government and private-sector cybersecurity colleagues for input on how to ensure that AI systems 


and ecosystems are secure and resilient to intelligent adversaries. Agencies involved in cybersecurity 
issues should engage their U.S. Government and private sector Al colleagues for innovative ways to 
apply AI for effective and efficient cybersecurity. 





AI in Weapon Systems 


The United States has incorporated autonomy into certain weapon systems for decades.” These 
technological improvements may allow for greater precision in the use of these weapon systems and safer, 
more humane military operations. Precision-guided munitions allow an operation to be completed with 
fewer weapons expended and with less collateral damage, and remotely-piloted vehicles can lessen the 
risk to military personnel by placing greater distance between them and danger. Nonetheless, moving 
away from direct human control of weapon systems involves some risks and can raise legal and ethical 
questions. The key to further incorporating autonomous and semi-autonomous weapon systems into U.S. 
defense planning and force structure is to continue ensuring that all our weapon systems, including 
autonomous weapon systems, are being used in a manner consistent with international humanitarian law. 
In addition, the U.S. Government should continue taking appropriate steps to control proliferation, and 
working with partners and Allies to develop standards related to the development and use of such weapon 
systems. 


Over the past several years, in particular, issues concerning the development of so-called “Lethal 
Autonomous Weapon Systems” (LAWS) have been raised by technical experts, ethicists, and others in 
the international community.” The United States has actively participated in the ongoing international 
discussion on LAWS in the context of the Convention on Certain Conventional Weapons (CCW),*° and 
anticipates continued robust international discussion of these potential weapon systems going forward. 


State Parties to the CCW are discussing technical, legal, military, ethical, and other issues involved with 
emerging technologies, although it is clear that there is no common understanding of LAWS. Some States 
have conflated LAWS with remotely piloted aircraft (military “drones’’), a position which the United 
States opposes, as remotely-piloted craft are, by definition, directly controlled by humans just as manned 
aircraft are. Other States have focused on artificial intelligence, robot armies, or whether “meaningful 
human control” — an undefined term — is exercised over life-and-death decisions. The U.S. priority has 
been to reiterate that all weapon systems, autonomous or otherwise, must adhere to international 


78 See, e.g., Jeffrey L. Caton, “Autonomous Weapons Systems: A Brief Survey of Developmental, Operational, 
Legal, and Ethical Issues,” Strategic Studies Institute, U.S. Army War College, December 2015, 
http://www.strategicstudiesinstitute.army.mil/pdffiles/P UB 1309.pdf. 


7 See, e.g., DeepMind comment, Human Rights Watch comment to the OSTP Request for Information on Artificial 
Intelligence. 


8° The Convention on Prohibitions on the Use of Certain Conventional Weapons Which May Be Deemed to Be 
Excessively Injurious or to Have Indiscriminate Effects (CCW) regulates certain weapons under its five Protocols. 
The States Parties to the CCW began informal discussions related to LAWS in 2014. 
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humanitarian law, including the principles of distinction*! and proportionality.” For this reason, the 
United States has consistently noted the importance of the weapons review process in the development 
and adoption of new weapon systems. The CCW will decide on whether and how to conduct future 
meetings on LAWS and associated issues during its Review Conference in December 2016. 


The U.S. government is also conducting a comprehensive review of the implications of autonomy in 
defense systems. In November 2012, the Department of Defense (DoD) issued DoD Directive 3000.09, 
“Autonomy in Weapon Systems,” which outlines requirements for the development and fielding of 
autonomous and semi-autonomous weapons. Weapon systems capable of autonomously selecting and 
engaging targets with lethal force require senior-level DoD reviews and approval before those weapon 
systems enter formal development and again before fielding. The DoD Directive neither prohibits nor 
encourages such development, but requires it to proceed carefully and only after review and approval by 
senior defense officials. Among other things, the DoD Directive requires that autonomous and semi- 
autonomous weapon systems are rigorously tested and that personnel are trained appropriately in their use 
to advance international norms pertaining to armed conflict. 


AI has the potential to provide significant benefits across a range of defense-related activities. Non-lethal 
activities such as logistics, maintenance, base operations, veterans’ healthcare, lifesaving battlefield 
medical assistance and casualty evacuation, personnel management, navigation, communication, cyber- 
defense, and intelligence analysis can benefit from AI, making American forces safer and more effective. 
AI may also play an important role in new systems for protecting people and high-value fixed assets and 
deterring attacks through non-lethal means. Ultimately, these applications may turn out to be the most 
important for DoD. 


Given advances in military technology and artificial intelligence more broadly, scientists, strategists, and 
military experts all agree that the future of LAWS is difficult to predict and the pace of change is rapid. 
Many new capabilities may soon be possible, and quickly able to be developed and operationalized. The 
Administration is engaged in active, ongoing interagency discussions to work toward a government-wide 
policy on autonomous weapons consistent with shared human values, national security interests, and 
international and domestic obligations. 


Recommendation 23: The U.S. Government should complete the development of a single, 
government-wide policy, consistent with international humanitarian law, on autonomous and 


semi-autonomous weapons. 





8! The principle of distinction requires parties to a conflict to distinguish between the civilian population and 
combatants and between civilian objects and military objectives, and to direct their operations only against military 
objectives. 


82 The principle of proportionality prohibits attacks that may be expected to cause incidental loss of civilian life, 
injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the 
concrete and direct military advantage anticipated. 
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Conclusion 


AI can be a major driver of economic growth and social progress, if industry, civil society, government, 
and the public work together to support development of the technology, with thoughtful attention to its 
potential and to managing its risks. 


Government has several roles to play. It should convene conversations about important issues and help to 
set the agenda for public debate. It should monitor the safety and fairness of applications as they develop, 
and adapt regulatory frameworks to encourage innovation while protecting the public. It should support 
basic research and the application of AI to public goods, as well as the development of a skilled, diverse 
workforce. And government should use AI itself, to serve the public faster, more effectively, and at lower 
cost. 


Many areas of public policy, from education and the economic safety net, to defense, environmental 
preservation, and criminal justice, will see new opportunities and new challenges driven by the continued 
progress of AI. Government must continue to build its capacity to understand and adapt to these changes. 


As the technology of AI continues to develop, practitioners must ensure that Al-enabled systems are 
governable; that they are open, transparent, and understandable; that they can work effectively with 
people; and that their operation will remain consistent with human values and aspirations. Researchers 
and practitioners have increased their attention to these challenges, and should continue to focus on them. 


Developing and studying machine intelligence can help us better understand and appreciate our human 
intelligence. Used thoughtfully, AI can augment our intelligence, helping us chart a better and wiser path 
forward. 
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Recommendations in this Report 


This section collects all of the recommendations in this report, for ease of reference. 


Recommendation 1: Private and public institutions are encouraged to examine whether and how they 
can responsibly leverage AI and machine learning in ways that will benefit society. Social justice and 
public policy institutions that do not typically engage with advanced technologies and data science in 
their work should consider partnerships with AI researchers and practitioners that can help apply AI 
tactics to the broad social problems these institutions already address in other ways. 


Recommendation 2: Federal agencies should prioritize open training data and open data standards in 
AI. The government should emphasize the release of datasets that enable the use of AI to address social 
challenges. Potential steps may include developing an “Open Data for AI” initiative with the objective of 
releasing a significant number of government data sets to accelerate AI research and galvanize the use of 
open data standards and best practices across government, academia, and the private sector. 


Recommendation 3: The Federal Government should explore ways to improve the capacity of key 
agencies to apply AI to their missions. For example, Federal agencies should explore the potential to 
create DARPA-like organizations to support high-risk, high-reward AI research and its application, much 
as the Department of Education has done through its proposal to create an “ARPA-ED,” to support R&D 
to determine whether AI and other technologies could significantly improve student learning outcomes. 


Recommendation 4: The NSTC MLAI subcommittee should develop a community of practice for AI 
practitioners across government. Agencies should work together to develop and share standards and 
best practices around the use of AI in government operations. Agencies should ensure that Federal 
employee training programs include relevant AI opportunities. 


Recommendation 5: Agencies should draw on appropriate technical expertise at the senior level when 
setting regulatory policy for Al-enabled products. Effective regulation of Al-enabled products requires 
collaboration between agency leadership, staff knowledgeable about the existing regulatory framework 
and regulatory practices generally, and technical experts with knowledge of AI. Agency leadership should 
take steps to recruit the necessary technical talent, or identify it in existing agency staff, and should 
ensure that there are sufficient technical “seats at the table” in regulatory policy discussions. 


Recommendation 6: Agencies should use the full range of personnel assignment and exchange models 
(e.g. hiring authorities) to foster a Federal workforce with more diverse perspectives on the current 
state of technology. 


Recommendation 7: The Department of Transportation should work with industry and researchers on 
ways to increase sharing of data for safety, research, and other purposes. The future roles of AI in 
surface and air transportation are undeniable. Accordingly, Federal actors should focus in the near-term 
on developing increasingly rich sets of data, consistent with consumer privacy, that can better inform 
policy-making as these technologies mature. 


40 


PREPARING FOR THE FUTURE OF ARTIFICIAL INTELLIGENCE 


Recommendation 8: The U.S. Government should invest in developing and implementing an advanced 
and automated air traffic management system that is highly scalable, and can fully accommodate 
autonomous and piloted aircraft alike. 


Recommendation 9: The Department of Transportation should continue to develop an evolving 
framework for regulation to enable the safe integration of fully automated vehicles and UAS, including 
novel vehicle designs, into the transportation system. 


Recommendation 10: The NSTC Subcommittee on Machine Learning and Artificial Intelligence 
should monitor developments in AI, and report regularly to senior Administration leadership about the 
status of AI, especially with regard to milestones. The Subcommittee should update the list of milestones 
as knowledge advances and the consensus of experts changes over time. The Subcommittee should 
consider reporting to the public on AI developments, when appropriate. 


Recommendation 11: The Government should monitor the state of AI in other countries, especially 
with respect to milestones. 


Recommendation 12: Industry should work with government to keep government updated on the 
general progress of AI in industry, including the likelihood of milestones being reached soon. 


Recommendation 13: The Federal government should prioritize basic and long-term AI research. The 
Nation as a whole would benefit from a steady increase in Federal and private-sector AI R&D, with a 
particular emphasis on basic research and long-term, high-risk research initiatives. Because basic and 
long-term research especially are areas where the private sector is not likely to invest, Federal 
investments will be important for R&D in these areas. 


Recommendation 14: The NSTC Subcommittees on MLAI and NITRD, in conjunction with the NSTC 
Committee on Science, Technology, Engineering, and Education (CoSTEM),, should initiate a study 
on the AI workforce pipeline in order to develop actions that ensure an appropriate increase in the size, 
quality, and diversity of the workforce, including AI researchers, specialists, and users. 


Recommendation 15: The Executive Office of the President should publish a follow-on report by the 
end of this year, to further investigate the effects of AI and automation on the U.S. job market, and 
outline recommended policy responses. 


Recommendation 16: Federal agencies that use Al-based systems to make or provide decision support 
for consequential decisions about individuals should take extra care to ensure the efficacy and fairness 
of those systems, based on evidence-based verification and validation. 


Recommendation 17: Federal agencies that make grants to state and local governments in support of 
the use of Al-based systems to make consequential decisions about individuals should review the terms 
of grants to ensure that Al-based products or services purchased with Federal grant funds produce 
results in a sufficiently transparent fashion and are supported by evidence of efficacy and fairness. 
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Recommendation 18: Schools and universities should include ethics, and related topics in security, 
privacy, and safety, as an integral part of curricula on AI, machine learning, computer science, and 
data science. 


Recommendation 19: AI professionals, safety professionals, and their professional societies should 
work together to continue progress toward a mature field of AI safety engineering. 


Recommendation 20: The U.S. Government should develop a government-wide strategy on 
international engagement related to AI, and develop a list of AI topical areas that need international 
engagement and monitoring. 


Recommendation 21: The U.S. Government should deepen its engagement with key international 
stakeholders, including foreign governments, international organizations, industry, academia, and 
others, to exchange information and facilitate collaboration on AI R&D. 


Recommendation 22: Agencies’ plans and strategies should account for the influence of AI on 
cybersecurity, and of cybersecurity on AI. Agencies involved in AI issues should engage their U.S. 
Government and private-sector cybersecurity colleagues for input on how to ensure that AI systems and 
ecosystems are secure and resilient to intelligent adversaries. Agencies involved in cybersecurity issues 
should engage their U.S. Government and private sector AI colleagues for innovative ways to apply AI 
for effective and efficient cybersecurity. 


Recommendation 23: The U.S. Government should complete the development of a single, government- 
wide policy, consistent with international humanitarian law, on autonomous and semi-autonomous 
weapons. 
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Acronyms 


AAAI 
AGI 

Al 
APEC 
BRAIN 
CALO 
CCC 
CCW 
CEA 
CEO 
CGC 
CoSTEM 
CS 
DARPA 
DoD 
DOT 
FAA 
FMVSS 
IARPA 
ICTs 
IPA 
LAWS 
MLAI 








Association for the Advancement of Artificial Intelligence 
Artificial General Intelligence 

Artificial Intelligence 

Asia-Pacific Economic Cooperation 

Brain Research through Advancing Innovative Neurotechnologies 
Cognitive Agent that Learns and Organizes 

Computing Community Consortium 

Convention on Certain Conventional Weapons 

Council of Economic Advisers 

Chief Executive Officer 

Cyber Grand Challenge (run by DARPA) 

Committee on Science Technology, Engineering, and Education (component of NSTC) 
Computer Science 

Defense Advanced Research Projects Agency 

Department of Defense 

Department of Transportation 

Federal Aviation Administration 

Federal Motor Vehicle Safety Standards 

Intelligence Advanced Research Projects Activity 

Information and Communication Technologies 

Intergovernmental Personnel Act 

Lethal Autonomous Weapon Systems 

Machine Learning and Artificial Intelligence (subcommittee of NSTC) 
National Airspace System 

National Economic Council 

National Highway Traffic Safety Administration 

National Institutes of Health 


Neural Information Processing Systems conference 


Z 


etworking and Information Technology Research and Development (subcommittee 
of NSTC) 


National Science Foundation 








National Science and Technology Council 


43 


PREPARING FOR THE FUTURE OF ARTIFICIAL INTELLIGENCE 


Organization for Economic Cooperation and Development 
Office of Management and Budget 

Office of Naval Research 

Office of Science and Technology Policy 

Research and Development 

Request For Information 

Science, Technology, Engineering, and Mathematics 
Unmanned Aerial System 


UAS Traffic Management 
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Abstract All of finance is now automated, most notably high frequency trading. 
This paper examines the ethical implications of this fact. As automation is an 
interdisciplinary endeavor, we argue that the interfaces between the respective 
disciplines can lead to conflicting ethical perspectives; we also argue that existing 
disciplinary standards do not pay enough attention to the ethical problems auto- 
mation generates. Conflicting perspectives undermine the protection those who rely 
on trading should have. Ethics in finance can be expanded to include organizational 
and industry-wide responsibilities to external market participants and society. As a 
starting point, quality management techniques can provide a foundation for a new 
cross-disciplinary ethical standard in the age of automation. 


Keywords High-frequency trading - Automation - Ethics - Quality management - 
Engineering 

Introduction 

In this article, we discuss ethics in finance in the age of automation rather than the 


ethics of automated trading. By “ethics”, we mean those special, morally 
permissible standards of conduct (rules, principles, or ideals) that apply to members 
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of a group simply because they are members of that group. Business ethics applies 
to people in business and no one else; legal ethics to lawyers and no one else; and so 
on. By “finance”, we mean (at a minimum) financial markets and financial 
services—equities and derivatives exchanges, market making, liquidity provision, 
and buy-side money management. Today, of course, high-frequency trading is a 
central part of finance so defined. 

Our goal is to provide a structured description of the evolving ethical landscape 
as finance tries to take account of how automation has changed its relation to 
external market participants and society. Automation has changed those whom 
trading can harm, how they might be harmed, and the scale of harm. (Harm here 
means both direct harm, in the sense of immediate financial loss, and indirect, in the 
sense of diminished confidence in financial markets.) Consider, for example, what 
happened to Knight Capital Group on August 1, 2012. The firm lost $440 million in 
less than 30 min when its new trading software flooded the market with orders, 
forcing the temporary closing of the New York Stock Exchange. The direct harm to 
the firm itself and its shareholders was catastrophic, almost bankrupting the firm 
(McCrank and Mollenkamp 2012). More indirect is the impact on the investing 
public’s confidence in the structure of financial markets. A loss of confidence may 
reduce investors’ appetite for risk and may thereby stall economic growth. None of 
that harm was necessary. Better release management, software operational controls, 
and testing, that is, a quality management system should have prevented it. 
Traditionally, the professions have treated preventing such harm to their clients, 
employers, and the public as a central ethical undertaking. 

Since a structured description of the evolving ethical landscape of finance 
invariably leads to ideas for new standards, we make some initial proposals, for 
example, for standards of quality that fill gaps between disciplines, gaps that now 
seem responsible for many of the financial disasters arising from automated trading. 


Background 


All financial markets and financial services are now automated: quantitative and 
fundamental research, data management, trade execution, and risk management. A 
macrocosm of this phenomenon is automated trading. In all its forms, including 
high-frequency trading, automation now accounts for more than 75 % of daily 





l! By “automated trading”, we mean any computerized system. We consider high-frequency trading to be 
a subset of automated trading. High-frequency trading systems include automated stock and option 
market-making systems, index arbitrage and equity long-short systems. Automated trading is a broader 
term encompassing high-frequency trading as well as longer term strategies such as statistical arbitrage, 
merger arbitrage, and execution systems for volume-weighted average price. What all automated trading 
systems have in common is that technology (computers, mathematical modeling, and the like) is a source 
of competitive advantage. 


? In this paper, we consider the automation of trading decisions and their execution in electronic markets. 
However, a subset or adjunct of automated trading (and just as important) is the use of automated 
algorithms that guide financial decisions that do not result in computer-generated executions, such as 
theoretical pricing and risk management algorithms. We do not address these forms of financial 
automation specifically, but recognize that our conclusions may apply to algorithms as well. 
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trading volume (Brogaard 2010). The advantage of automation to firms involved— 
proprietary trading and brokerage firms, hedge funds, clearing houses, exchanges— 
is the performance of tasks that are beyond human ability. Computers can perform 
financial calculations cheaper, faster, with (virtually) no mistakes across thousands 
of instruments and transactions (no matter how complex the math). 

Though not everyone agrees, most studies support the claim that automation 
benefits the overall functioning of both society and markets. Brogaard (2010) shows 
that high frequency trades and quotes “contribute more to price discovery than do 
non-[high-frequency trading] activity,” and that such trading reduces volatility; 
Jovanovic and Menkveld (2011), that high-frequency traders narrow the bid-ask 
spreads and may help unlock adverse-selection problems in markets; Hendershott 
et al. (2011), that automated trading improves liquidity; and Hendershott and 
Riordan (2011), that it makes markets more efficient. “The net benefit is that we 
have a better market with the participation of [high-frequency traders]” (Fabozzi 
et al. 2011). Automated trading tends to mitigate volatility, add liquidity, and reduce 
transaction costs for all market participants. 

There is, however, no longer any reason to doubt that automation can have bad 
effects. The question is why. One answer is that that high-frequency trading 
inevitably corrupts orderly markets, that automated systems only provide liquidity 
during normal markets, only to disappear during extreme events, thereby creating 
liquidity vacuums and exacerbating problems. There is reason to doubt this answer. 
Automated trading systems did not invent liquidity vacuums. They happened in 
human-driven markets, too. Pit traders walked off the floor in the 1987 crash. 
Likewise, brokers can refuse to answer the phone, and human market-makers can 
widen their bid-ask spreads. During normal market periods, competition aids in the 
creation of deep, liquid markets and price discovery. During times of market stress, 
society decries traders’ lack of a communal sense of obligation to the greater good. 

Kearns et al. (2010) suggest that this “debate over [automated trading] can be 
viewed as concern over the practices of one group of ‘players’ and the resulting 
costs to other players.” Thus, some have decried the potential for automated trading 
systems to game the market. “One smart computer may attempt to take advantage of 
the order-placement strategy of another not-so-smart computer with the possible 
effect that prices are driven away from equilibrium” (Stoll 2006). We call this kind 
of automated trading system “quality arbitrage”. High quality systems gain at the 
expense of low quality ones. Again, this is nothing new. Good traders have been 
making money off bad traders since markets began. And, surely, other systems will 
take advantage of disequilibria to make profit, forcing prices back into balance. As 
long as there is no systematic defrauding of other players (i.e. long-term investors), 





3 Quality arbitrage occurs when one automated market participant knows the automated (therefore, 
deterministic) behavior of other market participants—trading firm, clearing firm, exchange—and profits 
by exploiting that behavior. For example, if firm A’s trading system knows how firm B’s trading system 
will react under certain circumstances, firm A’s trading system can cause that circumstance to occur 
repeatedly and profit at the expense of firm B. Quality arbitrage can also occur if firms A and B use 
identical strategies, but gain market access through technologies with varying degrees of speed or quality. 
Further, if one exchange has inferior technology, a trading firm may be able exploit that inferiority 
relative to other exchanges/execution venues. Thus, at each point of trade selection and execution, quality 
may be the determining factor for sustainable competitive advantage. 
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it is hard to see how this activity (done right) could be construed as unethical (that 
is, aS a violation of an actual or desirable standard). 

Where once trading decisions were made based purely on human judgment, 
advances in technology and quantitative analysis enable mathematical models to 
approximate valuation and risk more closely. Nevertheless, these approximations 
have been tempered in the past by human oversight. The evolution to full 
automation of trading systems (especially, its speed) means that reaction to mis- 
approximations of value and risk can only be handled by computers within the 
prescribed decision structures of the software. Mis-approximations may occur 
because of inaccurate models, computer bugs, unstable designs, or lack of 
consideration of possible outcomes (i.e. “we didn’t know that could happen.”). 
Finance has evolved from human traders and manual control to automation and 
computer numerical control (Hassan et al. 2010). 

Automated trading systems lack human judgment, but their consequences can 
nevertheless be unethical (that is, the result of human conduct that, all things 
considered, should be prohibited). Moor (2006) calls such automatons “implicit 
ethical agents”. An implicit ethical agent is “programmed to behave ethically, or at 
least avoid unethical behavior, though without explicit representation of ethical 
principles’”—much as a safety catch on a gun may prevent unintended killing 
(Anderson and Anderson 2007). Trading systems implement rule-based decisions, 
essentially executing the ethics derived from their human designers. Thus, as 
emotionless agents, they break the traditional connection between human agency 
and social structure—the micro—macro link. See Von Scheve and Von Luede 
(2005). 

Ethics in financial services generally takes the form of “codes”, that is, 
systematic statements of standards of conduct (e.g. those of the Chartered Financial 
Analyst (CFA) (2008) and Certified Financial Planner (CFP) (2008)). These codes 
typically demand “integrity, objectivity, competence, fairness, confidentiality, 
professionalism and diligence.” (Ragatz and Duska 2010) Traders too have rules 
they must abide by, those of their exchange. However, the evolution to automation 
presents an opportunity to expand the role of ethics in finance beyond the domain of 
fiduciaries (i.e. money managers) and intermediaries (i.e. brokers and investment 
advisors) to include “financial engineers”* (i.e. mathematicians, usually called 
“quants”), software engineers, and computer engineers.” Automated finance brings 
together members of these different professions, but no one automatically or 
routinely takes responsibility for the consequences of their joint activity. “There is 
clearly an ethical imperative implicit in the growing influence of automation in 
market behavior. The ethical dimension of market automation is therefore worthy of 





4 “Financial engineering” is generally synonymous with mathematical finance, financial mathematics, 
quantitative finance, or computational finance. A survey of the codes of ethics of the relevant 
organizations—International Association of Financial Engineers, Professional Risk Managers Interna- 
tional Association, and the Global Association of Risk Professionals—uncovered no discussion of the 
ethics of financial automation. Programs in financial engineering are not accredited by ABET. Hence, the 
quotation marks around the term. Generally, “financial engineers” are not engineers strictly speaking. 


5 By “computer engineers”, we mean to include programmers, software quality engineers, and network 
engineers—many of whom will be computer scientists rather than engineers strictly speaking. 
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serious study.” (Hurlburt et al. 2009) Because automation enables virtually 
instantaneous trading between continents, that serious study should be global. The 
interface of the three different sets of professions (traders, quants, and computer 
engineers) gives rise to new ethical issues at the organizational (or system) level. 

In the age of automation, innovation—new strategies, methods of analysis, 
execution venues, instruments, technologies, data sources—are the ultimate source 
of competitive advantage. How is innovation to be kept socially useful? Given 
innovation’s rapid pace, neither governmental nor exchange regulation can keep up. 
Only ethics, the self-regulation of those closest to the innovation, can fill the gap 
between where innovation is and where governmental regulations and exchange 
rules suppose it to be. But, innovation’s pace also leaves little time for ethical 
reflection. As a new body of knowledge, the automated trading gold-rush has 
ignored the possibility of the self-organization of certain occupations into 
professions, the moment when codes of professional ethics generally appear (Davis 
1991). Thus, no code addresses ethical responsibilities in automated trading. None 
even requires (as engineering codes of ethics typically do) holding “paramount the 
safety, health, and welfare of the public.” (American Society of Civil Engineers 
(ASCE) 2011; American Society of Mechanical Engineers (ASME) 2011; and so 
on.) One of the problems in finance is that “people... believe that the task of 
creating and enforcing ethical rules and standards is the job of legislatures and 
regulators, not themselves.” (Boatright 2010). 


Competitive Advantage 


Parker (1998) argues that the challenge of globalization is the balancing of 
competing objectives such as ethics and competitive advantage. So, to understand 
that challenge, we must understand the sources of competitive advantage. Let us 
begin with a brief description of the three sources of competitive advantage in 
automated (or high-frequency) trading identified in Perkins et al. (2009) as carry- 
over from the traditional literature on finance: 


1. Strategic Creativity (or Differentiation): from an empirical perspective, the 
ability to perceive opportunities in a market that are unnoticed by human traders 
or other firms. 

2. Quantitative Improvement: the ability to capture profitable opportunities 
through more capable valuation and forecasting models, order management 
techniques, and trading logic. 

3. Data Cleaning and Optimization: the ability to uncover profitable strategies by 
repairing and manipulating financial data and optimizing signals. 


To this short list should be added five more sources of advantage: 


4. Order Execution Speed: the ability to respond more quickly to real-time market 
movements than market competitors. For example, given that two or more firms 
may become aware of the same arbitrage opportunity, only one firm, the firm 
with the fastest technology, can respond fast enough to capture that opportunity. 
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Fig. 1 Latency-cost trade-off 
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This creates an ethical problem, how to resolve the conflict between speed and 
the inclusion of fail-safe code that may add latency (that is, slowness in 
execution of strategy decision logic). 

Of course, achieving low latency costs money. Figure 1 shows the trade-off 
between latency and the launch cost of technology. The physical barrier is the 
geographical distance from the exchange, d, divided by the speed of light, c. This 
implies a trade-off between speed and context. There isn’t much time to think. 

5. Cost Minimization 


e Variable Cost Minimization (i.e. commission minimization or rebate 
maximization): the ability to execute strategic decisions more cheaply than 
competitors. That is, one firm may be able to execute a strategy profitably, 
while another firm may not be able to execute the same strategy profitably 
because of a higher commission or rebate structure. 

e Fixed Cost Minimization (i.e. minimizing fixed costs allocated to trading 
systems per accounting period): Building a trading infrastructure with direct 
market access alone could cost upwards of $15 million. Information 
technology, research and development, and overhead costs mean that one 
firm may be able to execute a strategy profitably, while another firm may 
not be able to do so because of higher fixed costs. An ethical conflict then 
exists between minimizing costs and satisfying obligations to foster a stable- 
market mechanism by, for example, paying for redundant systems. 


6. Rapid Model Deployment: the ability to manage trading-system research and 
development projects in order to quickly deploy capable systems on a 
technological platform. That is, because strategies are “perishable” (lose 
advantage quickly because of technical advances) and the value of a strategy is 
a function of the duration of expected cash flows, one firm may be able to 
capture more of a trading opportunity than other firms. 


Figure 2 shows that a strategy fails at time t. Thus, if deployment at time d 
occurs earlier, the cash-flow period t — d is longer, thereby increasing the present 
value of the strategy at t= 0. Of course, production pressure may lead to 
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Fig. 2 Time till failure, t, of a } I l 
trading strategy 
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circumvention of safety-driven project management processes and to the launch of a 
risky trading system, something that, all else equal, everyone involved in financial 
markets would like to avoid. 


7. Trading System Portfolio Management: the ability to optimize capital 
allocations across a group of working trading systems and the research and 
development of new systems. Portfolio management considers not only 
correlations of returns but also project and technology correlations. 

8. Defect Minimization (or Quality): not only do defects (such as software bugs or 
holes in the trading logic) reduce market return, but also investors’ confidence 
in the organization’s ability to succeed. The need for high quality in individual 
trading systems must, however, not take precedence over the overall quality— 
stability and reliability—of the global market system. 


We will hereafter refer to these sources of ethical problems by the names given 
here to the corresponding source of competitive advantage. Since regulators have 
begun to focus on automated trading, we shall now describe where regulation is 
(though it is evolving). We shall then describe the gap ethics can fill. 


Regulation 


Regulators in the US have addressed the ethical issues raised by market manipulation 
of automation (whether the manipulation was intentional, reckless, negligent, or 
merely accidental). So, for example, because ethics starts at the top (or, at least, is 
supposed to), Chief Executive Officers of automated trading firms must now annually 
certify the quality of their systems. The broadest regulation of electronic markets in 
the US is Reg. NMS (National Market System) of the security exchange commission 
(SEC). Its rules concerning order protection require traders and brokers to route 
client orders to the execution venue publishing the best price (such as an exchange or 
electronic communications network), not the venue offering the fastest execution or 
the greatest reliability. Some argue that the SEC has gone too far in managing 
business conduct, while others point out that Reg. NMS merely forces brokers to act 
in their client’ s best interests (Stoll 2006). Reg. NMS does not, however, specifically 
discuss any particular form of buy-side high-frequency or automated trading. 

SEC Rule 15c3-5 prohibits direct market access (DMA) to securities markets.° It 
requires trading firms connected to the exchange servers to establish and maintain 
risk-control systems and procedures. These systems must permit only electronic 
order requests (of the firm and its customers) that satisfy all pre-trade regulations. 
DMA precludes an intermediary (such as a broker-dealer or clearing member) from 





6 DMA refers to the facilities that allow a financial institution to route orders to the exchange order book 
without first passing through its clearing member or broker’s technology for pre-trade approvals. 
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Table 1 Market manipulation 





breakdown Root cause Manipulative outcome Prudence 
Intentional Reckless 
Strategy A B Q 
Technology C D 


interceding in the flow of order requests that a particular trading firm may generate. 
Thus, for DMA firms, there is a higher burden of responsibility to ensure the stable 
operation of their trading systems. For example, such firms must ensure that 
surveillance personnel receive real-time (or near real-time) post-trade exception 
reports. In this way, the regulation attempts to guarantee a level of quality control at 
the final link in the order entry chain. 

The regulations of the SEC and the Commodity Futures Trading Commission 
(CFTC) have also expressly prohibited market “manipulation”, though there is no 
definition of manipulation. Exchanges and courts are left to make their own 
decisions. What one exchange or one jury may consider manipulation, another may 
not, so arbiters must also consider the extent to which exchanges are complicit with 
strategies that occur on their electronic platforms. Recently, however, new 
regulations have lowered the bar for proving manipulation using automation (see 
SEC Rule 10b-5 and CFTC Rule 17 CFR Part 180.1). Where once the prosecutor’ s 
burden was to prove intent, now a merely manipulative outcome and proof of 
recklessness (i.e. conscious negligence or imprudence’) suffice to establish liability. 
The burden is on the accused trading firm to prove prudence in its trading system 
research, development, operations, and control processes. 

We may divide manipulation another way, into strategies and technology. 
Table 1 shows the resulting four categories of market manipulation. 

In box A of Table 1, a firm manipulates a market by intentionally developing a 
manipulative trading strategy. In box B, a firm deploys a reckless strategy in a 
trading system, though the technology is of high quality. Traders and quants design 
a strategy that, being poorly thought out, may behave in unexpected ways in certain 
market conditions. Programmers may be unaware of the potential outcomes of such 
a system. In box C, the strategy may be prudently researched and tested, but 
malicious technological implementation could introduce manipulation in its 
execution. For example, programmers could intentionally remove pre-trade 
safeguards to increase speed of order execution. This may violate the code of 
ethics governing programmers. In box D, reckless coding could introduce software 
bugs and lack of testing could permit deployment of a system that could go haywire. 
Box Q represents prudence in both strategy and technology R&D. Whatever the 
source of a manipulative outcome, the ethical failing was not in the real-time 
conscience of a trader, but rather in the weeks or even months of conscious research 
and development. But, what constitutes prudence, or good faith, in trading system 
research and development? This is where ordinary regulation ends, and the new 





7 Prudence has long been an ethical obligation in financial services. The Prudent Man Rule is an 
outgrowth of the landmark Harvard College v. Amory case in Massachusetts in 1830. 
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ethical territory begins: at the interface between quantified trading strategy and 
technological implementation. 

Exchanges and governments sometimes prohibit specific intentional strategic 
manipulation (i.e. box A). In addition to the traditional prohibitions—against, for 
example, front running, late trading by mutual funds, cornering the market, 
squeezing, pumping-and-dumping, market conditioning, and wash trading (see Kyle 
and Viswanathan 2008)—governmental regulations and exchange rules have 
banned or may ban certain intentional automation-enabled trading tactics. The 
most notable of these are flash trading, quote stuffing, and spoofing. Flash trading is 
the practice by which the exchange exposes a trade request to subscribers of the 
exchange’s data feed while not publishing the best price in the national market 
system. If any firm wishes to take the other side of the trade, the exchange can 
execute the trade (and keep the associated fees) before routing the order to another 
venue. Quote-stuffing is an attempt to overwhelm the network of an exchange by 
sending and cancelling thousands of trade requests per second, thereby blocking 
other market participants’ access to a market, a practice not only unfair but 
inconsistent even with an orderly market. Spoofing creates a false market, 
transmitting trade requests to an exchange with the express intent of cancelling 
them before execution, the goal being to create the illusion of a deep liquid market 
in order to fool other market participants or their systems. Exchanges have the 
ability to limit quote-stuffing and spoofing by fining firms based on a message-to-fill 
ratio, say 50-1. Firms that fail to execute at least one trade for every fifty messages 
(i.e. trade requests) sent to the exchange server are in violation. Some exchanges 
assess these fines, some don’t. Exchanges are then complicit in the rate of spoofing 
and quote-stuffing they allow. This presents an ethical question. What amount of 
messaging promotes a reasonably fair market, without infringing upon development 
of otherwise ethical trading strategies? 

Conceptually, spoofing is easily understood, but it is far harder to enforce 
regulations against it. In the Trillium Brokerage Services case, the Financial 
Industry Regulatory Authority was able to prosecute the firm successfully (Gordon 
2010). But consider options market-making, as an alternative example. As 
underlying prices change, market-makers continuously update their option bids 
and offers, though potentially never actually executing a trade. This well-understood 
strategy is not spoofing under any definition. Consider, on the other hand, a 
prudently-researched (i.e. box Q) trading strategy involving complex statistical 
relationships between multiple instruments. As prices in some markets change, the 
system may update bids and offers in other markets in a way not intuitive to 
regulators or individual exchanges. The messaging activity in any individual 
instrument may appear (in isolation) to be spoofing. Yet, the trading system is in fact 
in box Q. For any observed behavior, the question is: Is it driven by a box Q model, 
or is it spoofing? Is there, or is there not, a proven mathematical reason that the 
system is updating quotes and cancelling orders? Few firms would divulge such 
mathematical proof (the program’s logic). They prefer to fight a spoofing charge 
rather than give away their strategic creativity or quantitative improvements. 

Regulation may stifle competition, forcing a one-size-fits-all solution to such 
problems. Further, regulation in finance may result (and often has resulted) in 
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“regulatory capture” (the delegation of regulatory responsibility to the regulated). 
Because “self-regulation is generally more cost-effective than [governmental 
regulation]” (Lichtenberg, 1996), the industry has long established the individual 
exchanges and the Financial Industry Regulatory Authority as proxy regulators of 
their markets and members. Yet, even exchange regulations cannot cross national 
borders without significant international coordination. Only ethics (and similar 
guides to conduct) adopted by international bodies, such as the IEEE or the 
International Standards Organization (ISO), can adopt general principles and 
specific rules that cross national borders, market cultures, and even professional 
disciplines. 


Ethical Responsibilities 


Discussions of ethics in financial markets and financial services generally focus on 
professional responsibilities of money managers, brokers, investment advisors, and 
traders, who are bound by the codes of ethics of their respective professions as well 
as by governmental regulation and exchange rules. In the new age, however, the 
intermediaries are automated agents in a global automated mechanism (not 
members of any profession). As automation is an interdisciplinary endeavor, firms 
should also consider their organizational responsibilities to external stakeholders. 
As enablers of automated trading, the exchanges—NASDAQ, NYSE, CME, and so 
on—and even the off-exchange execution venues, should consider their ethical 
responsibilities (the tasks that should, in part at least, be guided by ethical 
standards). And, in fact, the exchanges, the Financial Industry Regulatory Authority, 
and their member firms are collectively and continually examining their industry- 
wide responsibilities to create an ethical climate, construct fair markets, and conduct 
themselves in a way that ultimately benefits society and helps to sustain confidence 
in the markets. There is, however, no single code of ethics or other mechanism 
defining the firm’s or the industry’s responsibilities to external stakeholders in the 
age of automated finance. 

Automated trading is complex. No one person knows all the strategies, all the 
quantitative methods, and all the technologies that produce competitive advantage. 
Teams of professionals—traders,* quants, and computer engineers—come together 
to move these systems from idea to operation. But, these professionals, as well as 
management, have conflicting perceptions and priorities. Each perceives the project 
through his or her own “intentions, interests, desires, points of view, or biases, all of 
which work as selective and restrictive filters.” (Werhane 1991) To appreciate the 
interactions between the functional areas, we generalize trading system R&D 
project management (in Stages 1-3) and O&C (in Stage 4) into four activities (see 
Kumiega and Van Vliet 2008): 


1. Design: Research into quantitative methods. The search for strategic differen- 
tiation and quantitative improvement invariably makes use of automated tools 





8 In the age of automated finance, the definition of trader is evolving. We do not pretend to know its 
future—or even its exact contours today. 
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Table 2 Four-stage R&D and O&C process 





Stages Trader Computer engineers Quants 

R&D 

1. Design Trading strategy Prototype testing Research, prototyping 
development 

2. Test Performance assessment Regression testing Data cleaning, optimization 

3. Implement Acceptance testing Technology development Quality assurance testing 

O&C 

4. Manage Real-time monitoring Technology control and Performance control 

refactoring 


(e.g. spreadsheets, statistical software packages, and the internet). Yet, these 
tools may create “ranks of sorcerer’s apprentice user-programmers...who have 
little training or expertise in how to avoid or detect high-risk...” (Boehm and 
Basili 2001). 

2. Test: Performance Testing (a.k.a. Backtesting). Data cleaning and parameter 
optimization involves backtesting to maximize some objective function, such as 
a Sharpe ratio (a measure of risk-adjusted performance). Yet, stability and 
safety is often unconsidered; such methods focus on profit, not safety or 
potential harm. 

3. Implement: Order Management and Trade Execution. Computer programs, 
hardware, and network platform encapsulate the strategy. The push for 
execution speed may neglect the stability of the firm’s systems or that of the 
execution venue. 

4. Management: Risk Management and Process Control. Performance monitoring 
should manage market and operational risks in real-time. But, pre- and post- 
trade controls are often viewed as adding too much latency, infringing on 
technological speed. 


The role of each functional area in these activities in shown in Table 2. 

Traders are traditionally (and often still) viewed as the primary drivers of profits, 
and often lead trading system research and development. They can push timelines, 
but they should not have the ability to override reliable data. They are the primary 
(but certainly not the only) source of strategic creativity. Encapsulation of their 
strategies into quantified logic—a process consisting of derivation and empirical 
testing of pricing, forecasting, risk management models, and trading logic—is 
usually performed by quants. Quants turn these algorithms over to programmers 
who implement them in computer code and optimize technological speed. There are 
interactions between the data, the math, and the technology. For example, speed 
may be traded off against numerical precision. 

Projects themselves have human components and therefore ethical consider- 
ations. The human aspects of trading systems are not in emotional reactions to 
market moves (as in the past), but rather in the individual reactions to various facets 
of management of trading-system projects. (See Kumiega and Van Vliet 2008, 2011 
for these facets.) Traders, quants, computer engineers, and management can view 


A Springer 


862 M. Davis et al. 


the same facts from different perspectives. Communication about trading-system 
problems has to overcome barriers to moving news (especially bad news) between 
traders, quants, and engineers, as well as up to management. It should hardly be 
surprising that such communications sometimes fail. All three disciplines should be 
aware of erring on the side of optimism in trading-system research and 
development. Fear of admitting intellectual failure can prove expensive, as 
demonstrated by the Challenger disaster (see Winsor 1988). Furthermore, past 
success can impact beliefs about the probability of future success. “Organizations 
often evolve gradually and incrementally into unexpected states.” (See Starbuck 
and Milliken 1988.) 

As in much of the project management and software engineering literature, ethics 
is embedded within discussions of methodology. (See PMI 2011; Rogerson and 
Gotterbarn 1998.) As it is the primary vehicle through which firms negotiate the 
tension between rapid model deployment, professional perspectives, and ethical 
obligations (Behnke 2009), rigorous methodology can prevent a normalization of 
deviance, prove prudence (i.e. Table 1, box Q), and reduce risk through objective 
criteria for project evaluation. As De Mast and Bisgaard (2007) point out: “if 
problems are not quantified, the trade-off nature is obscured and people tend to treat 
them as either/or problems and frequently politicize them.” A pre-condition of an 
ethical global trading mechanism, then, is that ethical cognitive processes drive 
trading-system research and development, and trump decisions driven by politics, 
fear, or greed. 

Overall organizational success is determined by the quality of the firm’s systems, 
not just the acumen of its individual traders. A brief review of ethical orientations of 
professionals, organizations, and the industry vis-a-vis the other stakeholders will 
frame further discussion of ethical problems. 


Professional Responsibilities 


Ideally, ethics are those morally permissible standards of conduct everyone in the 
relevant group (at their rational best) wants everyone else in the group to follow 
even if following them would mean having to do the same (Davis 2002). Ethics 
should guide behavior. But, human emotions can get in the way of both rational 
thinking and ethics. Emotions are part of the problem in trading, too. Fear and greed 
interfere with good decision-making. Lo et al. (2005) and Steenbarger (2002) show 
a negative correlation between emotional reactivity and successful trading. Lo and 
Repin (2002) show that “even the most seasoned trader exhibits significant 
emotional response...during certain transient market events such as increased price 
volatility or intra-day breaks in trend.” Emotional responses—fear and greed—tend 
to encourage ethical violations as well (see Sims 1992). Explicit ethical standards 
can remind people of the need to overcome fear and greed; license them to guide the 
conduct of others, whether by advice or criticism; and even justify procedures that 
make it harder to act on fear and greed. Hence, ethics has long been part of finance. 
But, each of the disciplines involved in automated trading has its own ethical 
climate, which affects how its members understand their respective roles and how 
they work with other disciplines. 
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Victor and Cullen (1988) identify five types of ethical climate by criteria for 
moral (including ethical) choice. An instrumental climate follows ethical egotism, 
often ignoring the interests of others. Caring climates consider others, but may 
provide justification for circumvention of organizational policies in order to help. 
Law and order climates rely on regulations, including ethical codes, without ethical 
reflection. Rules climates rely on internal policies and procedures irrespective of 
external norms (even ordinary morality). Independence climates permit individual 
reflection and action but provide little guidance. 


Traders 


Traders focus on markets and opportunities for profit. Of the rules and codes 
pertinent to traders (e.g. the regulations of the Chicago Mercantile Exchange, the 
National Futures Association, New York Stock Exchange, and the SEC), most 
developed in the age of face-to-face pit trading. (So, for example, some still prohibit 
smoking and fighting.) But, automated trading creates an anonymous environment 
without human interaction. Traditionally, the work climate among traders 
themselves has been instrumental and independent. This was easily justified 
because society benefited from their egoistic pursuit of their own interests. Which 
isn’t to say traders often behave badly. As liquidity providers, many take seriously 
their principal function of providing orderly markets. Yet, no rule or code applying 
to them specifically says anything about creating robust automated systems. 


Computer Engineers 


Computer engineers (including software engineers and computer scientists) focus on 
technological optimization. Computer engineers have their own codes of ethics which 
shape their professional climate. (See Association for Computing Machinery (ACM) 
(2011), IEEE (formerly, Institute of Electrical and Electronics Engineers) (2011), 
International Council on Systems Engineering (INCOSE) (201 1).) These codes require 
computer engineers to give special weight to the public interest, avoid unsafe 
practices, and accept prudent measures (see INCOSE 2011). The Software Engineer- 
ing Body of Knowledge (Abran et al. 2005) embeds ethics within the topic of software 
quality.” Kamthan (2008) points out that “the issue of information technology in 
general, and the role of quality in software development in particular, have been 
addressed in (Reynolds 2003; Tavani 2004). Moreover, software quality is viewed as 
an ethical issue (Peslak 2004).” As a body of knowledge, the focus on software quality 
and fail-safe design are part of both academic and professional software engineering. 
Ethics by way of software quality is taught in college and applied on the job. 


Quants 


Quants focus on mathematical constructs and data. Ethics in quantitative analysis is 
largely thought to be superseded by adherence to mathematical truth, exchange 





° There is also a Software Engineering Code of Ethics and Professional Practice (ACM/IEEE 1999), the 
result of a joint effort of the ACM and IEEE. 
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rules, and governmental regulation. Mathematical algorithms cannot (they feel) run 
afoul of regulations. While quants do have codes of ethics (see Professional Risk 
Managers’ International Association (PRMIA) (2011), American Statistical Asso- 
ciation (ASA) (2011)), they have little if any codified principles related to 
automation, though algorithms can be ethics-laden (see Kraemer et al. 2010). In the 
academy, quantitative finance is a discipline generally excluding normative 
questions (Boatright 2010). In practice, however, where implementation often 
converts “positive” research (that is, research where the goal is truth whatever the 
consequences) into normative automated algorithms (where the goal is profit), an 
ethical divide is crossed. Horrigan (1987) has traced this divide in the use of positive 
research to justify real-world decisions. For example, assumptions that need not 
conform to reality in a theoretical setting can take on potentially harmful 
implications in practice. Models that are put into a production environment need 
to be controlled. Further, Hurlburt et al. (2009) note that “when quants create and 
implement a model by focusing exclusively on short-term shareholder value, the 
social and economic consequences of the trades themselves might be completely 
ignored [allowing for great harm to be done].” 

Whose responsibility is it, then, to ensure that these systems are prudently 
designed and controlled? Professionals in each discipline may fail to recognize the 
ethical climates of the other two. Traders and quants are not bound by any code of 
ethics that binds computer engineers. Likewise, computer engineers may not grasp 
the ethical implications of the quantified algorithms they code or the market impact 
of the systems they design. They are not experts in finance, mathematics, or 
financial regulation. 

Automated trading systems cannot be designed and controlled without taking a 
stand on ethical issues. (See Kearns et al. (2010) for a similar conclusion.) 
Automation is an interdisciplinary endeavor. Traders, quants, and computer 
engineers each analyze data and perceive risk differently. While each profession 
may correctly perceive its own responsibilities in isolation, there are large ethical 
problems at the interfaces between them. These problems may be resolved by 
“organizational ethics”. 


Organizational Ethics 


“The ethical climate of an organization is the shared set of understandings about 
what is correct behavior and how ethical issues will be handled... When the ethical 
climate is not clear and positive, ethical problems will often result in unethical 
behavior.” (Sims 1992) In automated finance, the ethical agents are no longer self- 
interested people, but rather implicit ethical agents developed by self-interested 
organizations. Trading algorithms are fostering new organizational climates for the 
development, control, and protection of the firms’ intellectual property. 

In automated trading, it is the behavior of the interdisciplinary organization that 
really matters. The organization’s ethical problem is that the push for rapid model 
deployment conflicts with (time-consuming) backtesting, software quality testing, 
and consideration for the stability of the overall marketplace. Table 3 relates the 
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Table 3 Contributions to competitive advantage 





Sources of competitive Trader Quants Computer 

advantage programmers 

Professional Strategic Quantitative improvement, data cleaning Order execution 
creativity and optimization speed 

Organizational Rapid model deployment, cost minimization, portfolio management, defect 
minimization 


team’s activities to professional and organizational sources of competitive 
advantage. 

The ethics of automation (which is closely related to computer ethics, machine 
ethics, and the ethics of robotics) has recently emerged (Anderson and Anderson 
2006). Seminal contributions by Moor (2006), Allen et al. (2006), Ramaswamy and 
Joshi (2009) have given it structure. As a starting point, many authors point to the 
basic premise of Isaac Asimov’s Laws of Robotics (Asimov 1942): “a robot may 
not harm humanity, or, by inaction, allow humanity to come to harm.” If a trading 
system is to prevent harm, “it must be cognizant of possible harmful consequences 
and select its actions accordingly” (Allen et al. 2006). 

Ramaswamy and Joshi (2009) point out that automation enables unethical 
behavior and, furthermore, simplifies it by obscuring its source (“it wasn’t me, it 
was the computer, or the vendor’s software, or the other department.”). The 
presence of teams—where each member is bound (in theory) to a different 
professional standard—plus vendor-supplied components, creates plausible deni- 
ability for any individual. Programmers (bound in theory to the ethics of computer 
science) cannot foresee all possible evolutions of market prices and the reactions the 
system may take when functioning. “Hence the programmer cannot be held 
responsible if harm should be done as a secondary effect of the [system] interacting 
with [external market participants], so long as the [system] was not explicitly 
programmed to do harm... [Furthermore,] the modular design of systems can mean 
that no single person or group can fully grasp the manner in which the system will 
interact or respond to a complex flow of new inputs” (Allen et al. 2006). “This kind 
of blamelessness provides insufficient protection for those who might be harmed” 
(Allen et al. 2006). 

In addition, monitoring of market performance of working systems allows 
humans to adjust trading decision parameters in real-time. At what point and to what 
extent does the organization allow individuals to modify or overrule a machine or 
make on/off decisions?'° Much of this research shows that humans tend to over- 
correct and hence add to rather than subtract from risk. Like an amoral calculator 
(Kagan and Scholz 1984), the trading firm may have the attitude that as long as it 
can pay for its losses (whether caused by reckless design or over-correcting), the 


10 The engineering literature has examined behavioral aspects of systems monitoring and adjustment. 
Adaptive fail-safe controllers have proven to manage high-speed machines better than human controllers. 
The research of Suh and Cheon (2002) and Williams and Davies (1986) are notable in this regard. Bilson 
et al. (2010) and Hassan et al. (2010) have argued that an R&D process should prove stability and 
establish quantified limits for control of trading systems. 
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risks associated with trading-system failure are acceptable. If the trading firm settles 
at night (i.e. covers any margin requirements) with its clearing firms, it can trade the 
next day. Consideration of harm to external market participants has often been left 
out. But, regulation and ethics are changing this attitude. The ethical conflicts 
between professional climates are being resolved by new structures of organiza- 
tional responsibilities and a new industry-wide climate. 


Industry-Wide Responsibilities 


One problem for ethics in finance is that market outcomes (as with all outcomes) are 
uncertain and ambiguous. Ethical lapses may accidentally lead to good outcomes or 
vice versa, which can lead to “ethics in hindsight” (“no harm, no foul”). Yet, the 
best way to produce the social benefits of free markets is to foster market agents 
with good ethics. An industry-wide ethical climate can determine how organizations 
and professionals perceive problems, obligations to external stakeholders, and 
formal standards of right and wrong. 

The industry-wide climate has been “law and order”, driven by exchange rules 
and government oversight. Nevertheless, independence is still a widely held ideal. 
No industry-wide ethical climate should (it is felt) infringe upon a trading firm’s 
ability to innovate with new trading ideas and technologies. Any single prescriptive 
method would limit analysis and research, and hence restrict the efficiency of 
markets. Pluralism improves the critical power and evolution of markets. Some 
argue that automated trading’s high barriers to entry decrease this pluralism in the 
name of efficiency. But, automation has fragmented markets to such an extent that 
others decry the new existence of too much pluralism. 

In the age of automation, other cultural differences are complicating Hurlburt 
et al.’s (2009) imperative.'! Though virtually all exchanges seek the ethical goal of 
fair markets, the definition of fairness may vary. What is ethical in futures markets 
may be unethical in stock markets (e.g. in equities there are down-tick rules for 
short selling, in futures there is no such rule). Likewise, what is ethical in one 
country (or region) may be unethical in another. What is unethical in one of the 
functional disciplines may (as noted earlier) be un-addressed in another (and 
therefore ethical by default). Where individual markets, countries, organizations, 
and disciplines have their own rules, the new discussion must cross all such 
boundaries. Because these systems may trade across jurisdictions and countries, 
discussions in the industry have moved toward consistent regulations and shared 
understanding of how ethics should fill the gap. 


Ethical Problems 


Figure 3 summarizes our discussion thus far. The left-hand side depicts the 
traditional ethics of fiduciaries and intermediaries—money managers and invest- 
ment advisors. Their ethical guidelines are codified in exams for CFA, CFP, Series 





1 By culture, we simply mean a distinctive way of doing things. Not only nations and regions have 
cultures, but organizations, professions, and even occupations have them. 
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Automated Trading 
Ethics HFT Firm 





Fig. 3 Ethics in automated trading 


7, and so on. The right-hand side depicts trading-firm involvement in high- 
frequency trading (HFT). The HFT firm contains three functional areas each with its 
own ethics. Computer engineers’ ethics—derived from IEEE, ACM and INCOSE 
codes—focus on software testing and fail-safe mechanisms. Traders’ ethics derived 
from exchange rules traditionally focus on maintenance of orderly markets. Quant’s 
ethics—derived from professional organizations such as PRMIA and ASA—focus 
on mathematical truth and adherence to rules and regulations. The exchanges’ ethics 
focus on the operation of fair markets. 

Depicted in the lines connecting four ethical foci are the interfaces between them. 
This is where a new discussion can (and should) expand ethics in finance. To satisfy 
the ethical interface between disciplinary climates, other industries have turned to 
“quality”. Quality is not only a software testing or statistical framework, but also an 
ethical one (insofar as the standards in question are ones everyone in the relevant 
disciplines—at their rational best—wants everyone else in that discipline to follow 
even if that would mean having to do the same). The ethics literature has a long list 
of articles equating quality management and social responsibility. (See Tari (2011) 
for a comprehensive literature review.) Thus, quality may (at least in large part) fill 
the regulation-innovation gap and the interfaces of the disciplines in automated 
finance (depending on how “quality” is understood). 

The benefit to firms of understanding ethics as quality is that quality (generally) 
improves the bottom line. See, for example, Hendricks and Singhal (1997, 2000); 
Powell (1995); Flynn et al. (2007); Kaynak (2003); Chapman et al. (1997); 
Handfield et al. (1998). To be sure, many trading firms (and third-party vendors of 
trading-system components) understand this. They perform rigorous quality control 
over their processes—not strictly out of ethics, but out of an understanding that a 
quality approach to innovation is, in the end, the source of competitive advantage. 
However, even these systems can fail at the disciplinary interfaces because 
methodology has not been coordinated. What we would like to do now is address 
particular issues in financial automation that require cross-disciplinary ethical 
decisions that discipline-based concepts of quality do not. 
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The Flash Crash 


The flash crash of May 6, 2010 illustrates the value-ladeness of a trading firm’s 
decision to stop trading and the variety of ethical perspectives now interacting. 
Some automated trading firms shut down their systems because they thought it was 
the ethical thing to do. After all, something seemed to have gone wrong with the 
market, and profiting from operational failure of its infrastructure would be 
unethical. (The motivation behind this reasoning is the fail-safe ethics of computer 
engineers.) Others chose to leave their machines on because they thought it was the 
ethical thing to do. After all, liquidity vacuums create volatility and perhaps panic, 
and the best we can do in a suddenly contracting market is to continue to provide 
liquidity, albeit at safe prices. (The motivation behind this reasoning is traders’ 
ethics, a focus on orderly markets.) 

Liquidity providers had two options: (1) shut down their systems until the 
problems were rectified, at which time they would reboot and reconnect to the 
exchange servers, a process which could take at least 20 min; and (2) continue to 
provide quotes to the marketplace, though with wider bid/ask spreads to protect the 
firm (i.e. say bids of one cent instead of the forty of a few minutes before), so that 
when problems were resolved, real bids and offers could be posted immediately. Is it 
fair that (even ethically-minded) liquidity providers bought stock at one cent? 

Some exchanges sought to rectify (what they considered) unfairness by “busting 
trades” in subsequent days. But, some liquidity providers had already taken other 
positions in other markets to reduce risk. The original trades (made in good faith) were 
hedged in other markets. Busting the initial trade after the fact left firms’ subsequent 
positions unhedged, leading to potentially large losses. Going forward, liquidity 
providers would have a disincentive to continue trading when the risk of such a 
scenario arose again. The incentive is to pull liquidity as soon as there is any sign of 
market stress. This would surely add to the stress by causing a liquidity vacuum. 

Contrast this policy of busting trades to the response of the Chicago Mercantile 
Exchange (CME) to an exchange testing mistake in September, 2010, which 
accidentally sent 30,000 orders onto its live electronic platform. The orders left 
some traders with positions they did not expect. The CEO and President of the CME 
quickly issued a written apology, but in deference to trades placed in good faith, did 
no busting. Rather, the CME “work[ed] with all affected customers to neutralize the 
financial effect of these transactions, including making customers whole for net 
losses they incurred in exiting positions.” (Lynch 2010) (By “net losses” they 
meant to include losses from subsequent hedging transactions.) This response seems 
better, since it fits the interface between traders’ ethics and the exchange’s ethics— 
on Fig. 3, the fairness-orderliness interface. Nevertheless, there is no industry-wide 
agreement as to what standard should prevail in future situations. So, the lack of 
agreement on a standard at the interfaces could still lead to a crash. 

Of course, some of these problems will correct themselves. Today’s murky 
ethical waters may tomorrow be seen merely as missed opportunities through the 
clarity of historical testing of new automated strategies (backtesting). Market data 
becomes part of the historical record. New strategies to exploit a particular extreme 
circumstance will tend to reduce the probability of its recurrence. 
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While SEC Rule 15c3-5 prohibiting direct market access defines organizational 
responsibilities for controlling order entry to electronic exchanges, it falls short of 
fostering an ethical industry-wide climate. So, according to the principles of the 
American Society for Quality, the risk of failure still exists. To make matters worse, 
the SEC rule creates a false sense of security, only shifting the burden of compliance 
from trading firms to brokers, under the assumption that brokers’ systems are of higher 
quality. So, the ethical problem remains as well. Viewed from an ethical standpoint, 
direct market access touches on all the interfaces in Fig. 3. To satisfy all the ethical 
perspectives, trading firms must have a way to test the fail-safeness of their 
technologies and the robustness of their quantified strategies. One way to test is to use 
the world (live trading). That is much like flying experimental planes over crowded 
cities to test their airworthiness. Much better for testing would be some simulation or 
laboratory. Yet, such safe venues for testing rarely exist in finance—or, at least, rarely 
exist in a form sufficientldevelopment, conflicts can occur between disciplin 

y robust to serve in place of live trading. To satisfy the ethical obligations at the 
fairness-fail-safe interface between a firm’s computer engineers and the exchange, 
software-quality techniques address methods for testing exchange network 
gateways. But, without full testing environments and defined acceptance tests, 
strategies cannot be vetted safely. Thus, neither the fairness-rule interface between 
quant and exchange ethics, nor the orderliness-fairness interface between traders 
and exchange ethics, can be satisfied by current regulations. 

Exchanges and execution venues in coordination with trading firms and clearing 
members have (or at least should have) an ethical obligation to create test 
environments and standards for system run-off tests. These should bombard each 
system with all manner of market and technological scenarios to verify its 
robustness. The availability of such venues would provide a means for all 
professionals and organizations to satisfy their own obligations. This would fix the 
problem, not just contain it as the current solution does. Better regulation could 
encourage an industry-wide ethical climate through mandatory inter-firm coordi- 
nation for development of these venues. In the end, it is the quality of the industry’s 
interconnected systems that will determine fairness, the ultimate ability of markets 
to benefit society and sustain confidence in the markets. 


Project Management 


Figure 3 shows the interfaces between professionals, which can cause inappropriate 
decisions to be made. Each professional analyzes data and risk from his own 
profession’s perspective and considers its own priorities. From a project manage- 
ment framework, rigorous inter-disciplinary methodology and management check 
points—often called Stage-Gates' (see Cooper 2001)—in research and develop- 
ment can align professional ethics with the ethics of the exchange. 

Success in automated trading depends on success of the firm’s business model, 
which demands rapid model deployment. Scarce resources lead to production 
pressure and decisions to launch otherwise risky trading systems. Managers of 
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research and development for trading systems are forced to consider tradeoffs 
between ethics (including quality standards) and the bottom-line. Within research 
and development, conflicts can occur between disciplines. Is there a difference 
between thinking like a trader and thinking like a computer engineer?" In 
automated trading, there is. If a trader or quant tells a programmer not to test her 
code, neither has committed any violation with respect to the relevant code of ethics 
(that is, the trader’s code, the quant’s code, or the programmer’s code). Each has this 
plausible deniability in the event of harm. Each side can argue it was ethical. Poor 
project management across the three professional areas and the lack of robust test 
platforms leads inevitably to boxes B, C, and D in Table 1. The existence of 
schedule-minded, amoral calculations among top management may violate safety 
norms protecting external market participants. For management, profit goals often 
have priority over the safety of the global trading mechanism (at least when risks 
seem low). The industry needs a holistic framework, where ethical problems can be 
addressed before management has begun to worry about schedule. 


Quality Climate 


Quality processes implements an organization’s ethics (in part at least). In the 
financial industry, discussion of quality focuses on specific guidance for improve- 
ment of trading systems within the ethical silos of the respective professions. But a 
focus on quality can also avoid compromising trader independence, methodological 
pluralism, and disclosure of proprietary sources of competitive advantage. This is 
why in Table 1, Q stands for quality. Beyond simple software testing, quality in 
finance can include all professionals—computer engineers, quants, and traders alike. 

The ISO 9000 family of standards, and its industry-specific derivatives, are the 
most widely recognized quality management system standards. In other industries 
(where public safety must be ensured), ISO standards define how firms ought to do 
business in, for example, aerospace (which operates under its industry-specific 
standard AS 9000), chemicals (The Dow Chemical Company Quality Management 
System Manual, for instance), medical devices (which uses its own ISO 13485), and 
food safety (which has its own, ISO 22000). 

A similar standard for finance should include software and hardware testing that 
proves the firm has demonstrated that its automated trading systems function 
properly, are operationally safe, and are robust enough to behave acceptably during 
potential extreme events. 

The ability of automated trading firms to prove the stability of their systems 
depends upon the availability of execution-venue-simulation facilities to fully test 
those systems. Such simulation facilities should provide testing against all manner 
of extreme market and infrastructure events. 

By following ISO 9000-style standards, an automated trading firm can satisfy its 
organizational obligations to prove and document that trading strategies and 





12 Mason asked Lund to think like a manager rather than an engineer before Lund approved the launch of 
the Challenger space shuttle after initially opposing it as Vice President of Engineering (Davis 1991) 
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technologies will operate safely, reliably, and profitably. There is a large literature 
demonstrating that the use of quality management systems increases the financial 
performance of the firm. 

But, how can the government get high-frequency trading firms to improve their 
quality climate? Since one of top management’s responsibilities now is to ensure 
that both strategies and technologies are researched, developed, and controlled 
according to quality standards (whoever else may share in that responsibility), 
governmental regulation should hold management accountable for the failure of 
employees to design quality in. This may be done by some combination of criminal 
and civil liability. 

These standards should mean the financial industry is taking the lead role in 
improving its own management systems. The federal government may then be less 
likely to issue prescriptive regulation, allowing it to concentrate on monitoring the 
integrity of the trading organization as opposed to the details of individual trading 
systems compliance. (That is how regulators have responded to aerospace’s AS 
9000.) Likewise, global industry standards should present significant opportunities 
for regulatory harmonization across national boundaries. 


Conclusion 


The Challenger disaster was caused by pressure to launch, communication 
breakdowns between engineers and management, and differing perspectives and 
priorities (Werhane 1991). To reduce the likelihood of a breakdown in the global 
trading mechanism in the age of automation, the financial industry needs a holistic 
cross-cultural ethical framework to fill the gap between innovation and regulation. 
Properly framing the ethical problems will enable root-cause analysis and long-term 
fixes, rather than short-term regulatory containments of undesirable outcomes. In 
this age of automation, quality is (or should be) a professional, organizational, and 
industry-wide obligation. Even optimal professional ethics (in isolation) will not 
provide adequate protection for those who might be harmed by cross-disciplinary 
failures of trading systems or the global trading mechanism. A broad ethical 
framework must ensure prudence without infringing on the necessary independence 
of trading firms. 

As a Starting point, quality management techniques can provide a structure for 
understanding and solving the inter-disciplinary ethical problems. Then quality 
techniques can be employed to find solutions to root causes rather than (the 
industry’s current mode) containment of individual outcomes. Rigorous quality 
methods can prove good faith and reduce risk through objective criteria for project 
evaluation. Since all of finance is (more or less) automated, these concepts could be 
extended to areas external to automated trading, such as risk and portfolio 
management. 

Regulators can encourage ethics through quality mandates that focus on the 
whole system, not individual professions, disciplines, or organizations. A project 
management framework that includes stage-gates could reduce the potential risks 
individual trading systems may pose to the global trading mechanism. Given formal 
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decision-making structures for organizational responsibility, regulators can assess 
the rigor of research, development, operations, and control without infringing on the 
secrecy of proprietary trading strategies and their sources of competitive advantage. 
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Executive Summary 


Artificial intelligence (AI) and machine learning are being rapidly adopted for a range of 
applications in the financial services industry. As such, it is important to begin considering the 
financial stability implications of such uses. Because uses of this technology in finance are in a 
nascent and rapidly evolving phase, and data on usage are largely unavailable, any analysis 
must be necessarily preliminary, and developments in this area should be monitored closely. 


Many applications, or “use cases”, of AI and machine learning already exist. The adoption of 
these use cases has been driven by both supply factors, such as technological advances and the 
availability of financial sector data and infrastructure, and by demand factors, such as 
profitability needs, competition with other firms, and the demands of financial regulation. Some 
of the current and potential use cases of AI and machine learning include: 


- Financial institutions and vendors are using AI and machine learning methods to assess 
credit quality, to price and market insurance contracts, and to automate client 
interaction. 


- Institutions are optimising scarce capital with AI and machine learning techniques, as 
well as back-testing models and analysing the market impact of trading large positions. 


- Hedge funds, broker-dealers, and other firms are using AI and machine learning to find 
signals for higher (and uncorrelated) returns and optimise trading execution. 


- Both public and private sector institutions may use these technologies for regulatory 
compliance, surveillance, data quality assessment, and fraud detection. 


With the FSB FinTech framework,’ our analysis reveals a number of potential benefits and 
risks for financial stability that should be monitored as the technology is adopted in the coming 
years and as more data becomes available. In some cases, these observations are also contained 
in the FSB report on regulatory and supervisory issues around FinTech.” They are: 


- The more efficient processing of information, for example in credit decisions, financial 
markets, insurance contracts, and customer interaction, may contribute to a more 
efficient financial system. The RegTech and SupTech applications of AI and machine 
learning can help improve regulatory compliance and increase supervisory 
effectiveness. 


- Atthe same time, network effects and scalability of new technologies may in the future 
give rise to third-party dependencies. This could in turn lead to the emergence of new 
systemically important players that could fall outside the regulatory perimeter. 


- Applications of AI and machine learning could result in new and unexpected forms of 
interconnectedness between financial markets and institutions, for instance based on the 
use by various institutions of previously unrelated data sources. 


' FSB (2016), “Fintech: Describing the Landscape and a Framework for Analysis,” March [unpublished]. 


2 FSB (2017), “Financial Stability Implications from FinTech, Supervisory and Regulatory Issues that Merit Authorities 
Attention,” June. 


- The lack of interpretability or “auditability” of AI and machine learning methods could 
become a macro-level risk. Similarly, a widespread use of opaque models may result in 
unintended consequences. 


- As with any new product or service, there are important issues around appropriate risk 
management and oversight. It will be important to assess uses of AI and machine 
learning in view of their risks, including adherence to relevant protocols on data privacy, 
conduct risks, and cybersecurity. Adequate testing and ‘training’ of tools with unbiased 
data and feedback mechanisms is important to ensure applications do what they are 
intended to do. 


Overall, AI and machine learning applications show substantial promise if their specific risks 
are properly managed. The concluding section gives preliminary thoughts on governance and 
development of models, as well as auditability by institutions and supervisors. 


Introduction 


This report analyses possible financial stability implications of the use of artificial intelligence 
(AJ) and machine learning in financial services. It was drafted by a team of experts from the 
FSB Financial Innovation Network (FIN). The report draws on discussions with firms; 
academic research; public and private sector reports; and ongoing work at FSB member 
institutions.* The report analyses potential financial stability implications of the growing use of 
AI by financial institutions. Given the relative novelty of many applications, and the paucity of 
data on adoption, it is necessarily a horizon-scanning piece. 


The report is structured as follows. In section 1, the key concepts of the report are defined, and 
some background is given on the development of AI and machine learning for financial 
applications. Section 2 describes supply and demand factors driving the adoption of these 
techniques in financial services. Section 3 describes four sets of use cases: (1) customer-focused 
applications; (ii) operations-focused uses; (iii) trading and portfolio management; and (iv) 
regulatory compliance and supervision. Section 4 contains a micro-analysis of the effects of 
adoption on financial markets, institutions and consumers. Section 5 gives a macro-analysis of 
effects on the financial system. Finally, section 6 concludes with an assessment of implications 
for financial stability. 


1. Background and definitions 


Researchers in computer science and statistics have developed advanced techniques to obtain 
insights from large disparate data sets. Data may be of different types, from different sources, 
and of different quality (structured and unstructured data). These techniques can leverage the 
ability of computers to perform tasks, such as recognising images and processing natural 
languages, by learning from experience. The application of computational tools to address tasks 
traditionally requiring human sophistication is broadly termed ‘artificial intelligence’ (AI). As 
a field, AI has existed for many years. However, recent increases in computing power coupled 
with increases in the availability and quantity of data have resulted in a resurgence of interest 
in potential applications of artificial intelligence. These applications are already being used to 
diagnose diseases, translate languages, and drive cars; and they are increasingly being used in 
the financial sector as well. 





FIN held two workshops on this topic. The first workshop was held on 4 April 2017 in San Francisco. The second workshop 
was held on 27 June 2017 in Basel. The participants at these workshops included representatives from 7 financial 
institutions, six artificial intelligence firms, three large tech firms and two industry organisations from North America, 
Europe and Asia. In addition, drafting team members and the FSB secretariat conducted bilateral conversations with 
relevant private sector contacts across a range of jurisdictions. 


The report draws on some examples from specific private firms involved in FinTech. These examples are not exhaustive 
and do not constitute an endorsement by the FSB for any firm, product or service. Similarly, they do not imply any 
conclusion about the status of any product or service described under applicable law. Rather, such examples are included 
for purposes of illustration of new and emerging business models in the markets studied. 


Various financial regulatory authorities have defined the big data phenomenon as a confluence of factors, including the 
ubiquitous collection of data from a variety of sources, the plummeting cost of data storage and powerful capacity to analyse 
data. See, e.g., U.S. Federal Trade Commission Report, January (2016,), “Big Data: A tool for Inclusion or Exclusion?” 
January, p. 1; EBA, EIOPA and ESMA (2016), “European Joint Committee Discussion Paper on the Use of Big Data by 
Financial Institutions,” JC 2016 86, p. 7. 


There are many terms that are used in describing this field, so some definitions are needed 
before proceeding. ‘Big data’ is a term for which there is no single, consistent definition, but 
the term is used broadly to describe the storage and analysis of large and/or complicated data 
sets using a variety of techniques including AI.° Analysis of such large and complicated datasets 
is often called “big data analytics.” A key feature of the complexity relevant in big data sets 
analytics often relates to the amount of unstructured or semi-structured data contained in the 
datasets. 


This report defines AI as the theory and development of computer systems able to perform tasks 
that traditionally have required human intelligence. AI is a broad field, of which ‘machine 
learning’ is a sub-category.’ Machine learning may be defined as a method of designing a 
sequence of actions to solve a problem, known as algorithms,® which optimise automatically 
through experience and with limited or no human intervention.’ These techniques can be used 
to find patterns in large amounts of data (big data analytics) from increasingly diverse and 
innovative sources. Figure 1 gives an overview. 


Figure 1: A schematic view of AI, machine learning and big data analytics 
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Many machine learning tools build on statistical methods that are familiar to most researchers. 
These include extending linear regression models to deal with potentially millions of inputs, or 
using statistical techniques to summarise a large dataset for easy visualisation. Yet machine 


6 Jonathan Stuart Ward and Adam Barker (2013), “Undefined By Data: A Survey of Big Data Definitions” Cornell University, 
arXiv:1309.5821. 





Examples of AI applications that are not machine learning include the computer science fields of ontology management, 
or the formal naming and defining of terms and relationships by computers, as well as inductive and deductive logic and 
knowledge representation. In this report, for completeness, we often refer to “AI and machine learning,” with the 
understanding that many of the important recent advances are in the machine learning space. 


An algorithm may be defined as a set of steps to be performed or rules to be followed to solve a mathematical problem. 
More recently, the term has been adopted to refer to a process to be followed, often by a computer. 


° Arthur Samuel (1959), “Some Studies in Machine Learning Using the Game of Checkers,” IBM Journal: 211-229; Tom 
Mitchell (1997), Machine Learning, New York: McGraw Hill; Michael Jordan and Tom Mitchell (2015), “Machine 
learning: Trends, perspectives, and prospects,” Science 349(6245): 255-260. Samuel defined machine learning as the “field 
of study that gives computers the ability to learn without being explicitly programmed,” while Mitchell defines it as the 
“the question of how to build computers that improve automatically through experience.” 


learning frameworks are inherently more flexible; patterns detected by machine learning 
algorithms are not constrained to the linear relationships that tend to dominate economic and 
financial analysis. In general, machine learning deals with (automated) optimisation, prediction, 
and categorisation, not with causal inference. !° In other words, classifying whether the debt of 
a company will be investment grade or high yield one year from now could be done with 
machine learning. However, determining what factors have driven the level of bond yields 
would likely not be done using machine learning. 


There are several categories of machine learning algorithms. These categories vary according 
to the level of human intervention required in labelling the data: 


e In ‘supervised learning’, the algorithm is fed a set of ‘training’ data that contains labels 
on some portion of the observations. For instance, a data set of transactions may contain 
labels on some data points identifying those that are fraudulent and those that are not 
fraudulent. The algorithm will ‘learn’ a general rule of classification that it will use to 
predict the labels for the remaining observations in the data set. 


e ‘Unsupervised learning’ refers to situations where the data provided to the algorithm 
does not contain labels. The algorithm is asked to detect patterns in the data by 
identifying clusters of observations that depend on similar underlying characteristics. 
For example, an unsupervised machine learning algorithm could be set up to look for 
securities that have characteristics similar to an illiquid security that is hard to price. If 
it finds an appropriate cluster for the illiquid security, pricing of other securities in the 
cluster can be used to help price the illiquid security. 


e ‘Reinforcement learning’ falls in between supervised and unsupervised learning. In this 
case, the algorithm is fed an unlabelled set of data, chooses an action for each data point, 
and receives feedback (perhaps from a human) that helps the algorithm learn. For 
instance, reinforcement learning can be used in robotics, game theory, and self-driving 
cars. !! 


e ‘Deep learning’ is a form of machine learning that uses algorithms that work in ‘layers’ 
inspired by the structure and function of the brain. Deep learning algorithms, whose 
structure are called artificial neural networks, can be used for supervised, unsupervised, 
or reinforcement learning. 


Recently, deep learning has led to remarkable results in diverse fields, such as image 
recognition and natural language processing (NLP). Deep learning algorithms are capable of 
discovering generalisable concepts, such as encoding the concept of a ‘car’ from a series of 
images. An investor might deploy an algorithm that recognises cars to count the number of cars 
in a retail parking lot from a satellite image in order to infer a likely store sales figure for a 
particular period. NLP allows computers to ‘read’ and produce written text or, when combined 


Here, prediction is understood as identifying something as likely before the event based on experience. Causal inference 
and forecasting are done from a scientific perspective on the basis of analysis of the past. 


This categorisation of machine learning algorithms is taken from Kolanovic, Marko and Rajesh Krishnamachari (2017), 
“Big Data and AI Strategies: Machine Learning and Alternative Data Approach to Investing,” JP Morgan, May; and Internet 
Society (2017), “Artificial Intelligence and Machine Learning: Policy Paper,” Internet Society White Paper, April. 
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with voice recognition, to read and produce spoken language. This allows firms to automate 
financial service functions previously requiring manual intervention. 


Machine learning can be applied to different types of problems, such as classification or 
regression analysis. Classification algorithms, which are far more frequently deployed in 
practice, group observations into a finite number of categories. Classification algorithms are 
probability-based, meaning that the outcome is the category for which it finds the highest 
probability that it belongs to. An example might be to automatically read a sell-side report and 
label it as ‘bullish’ or ‘bearish’ with some probability, or estimate an unrated company’s initial 
credit rating. Regression algorithms, in contrast, estimate the outcome of problems that have an 
infinite number of solutions (continuous set of possible outcomes). This outcome can be 
accompanied with a confidence interval. Regression algorithms can be used for the pricing of 
options. Regression algorithms can also be used as one intermediate step of classification 
algorithm. 


It is important to note what machine learning cannot do, such as determining causality. 
Generally speaking, machine learning algorithms are used to identify patterns that are correlated 
with other events or patterns. The patterns that machine learning identifies are merely 
correlations, some of which are unrecognisable to the human eye. However, AI and machine 
learning applications are being used increasingly by economists and others to help understand 
complex relationships, along with other tools and domain expertise. 


Many machine learning techniques are hardly new. Indeed, neural networks, the base concept 
for deep learning, were first developed in the 1960s.!? However after an initial burst of 
excitement, machine learning and AI failed to live up to their promises and funding dissipated 
for over a decade, in part because of the lack of sufficient computing power and data. There 
was renewed funding and interest in applications in the 1980’s, during which many of the 
research concepts were developed for later breakthroughs. ! 


By 2011 and 2012, driven by the vast increase in the computational power of modern 
computers, machine learning algorithms, especially deep learning algorithms, began to 
consistently win image, text, and speech recognition contests. Noticing this trend, major tech 
companies began to acquire deep learning start-ups and rapidly accelerate deep learning 
research.'* Also new is the scale of collection of big data, for example the ability to capture 
data on the scale of every single credit card transaction or every word on the web, and even 
‘mouse’ hovers over websites. Other advances have also helped, such as increased 
interconnectedness of information technology resources with cloud computing architecture, 
with which big data can now be organised and analysed. Using data sets of this size and 
complexity and with the increase in computing power, machine learning algorithms results have 
improved, some of which are highlighted in the sections that follow. This has also spurred large 





2 The ‘K-Nearest-Neighbour’ algorithm, which is the simple, intuitive machine learning algorithm taught at the start of many 
undergraduate Computer Science classes, was proposed in its modern form in 1967. See Thomas Cover and Peter Hart 
(1967), “Nearest Neighbor Pattern Classification,” JEEE transactions on information theory. 13.1: 21-27. 


13 See Luke Dormehl (2016), Thinking Machines: The Quest for Artificial Intelligence--and Where It's Taking Us Next, 
London: Penguin Books. 


Tim Dettmers (2015), “Deep Learning in a Nutshell: History and Training,” Parallel Forall, December. Accessed on May 
30, 2017. Web. https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/ 





investments in AI start-ups. The World Economic Forum reported that global investment in AI 
start-ups rose from $282 million in 2011, to $2.4 billion in 2015.'° The number of merger and 
acquisition (M&A) deals in AI has also accelerated over this period (figure 2). 


Figure 2: Global artificial intelligence merger and acquisition activity, 2012-2017 





Source: CB Insights (2017), “The Race For AI: Google, Baidu, Intel, Apple In A Rush To Grab Artificial Intelligence Startups,” 
Research Brief, July. 


Many applications tend more toward ‘augmented intelligence,’ or an augmentation of human 
capabilities, rather than a replacement of humans. Even as advancements in AI and machine 
learning continue, including in the area of deep learning, most industries are not attempting to 
fully replicate human intelligence. As noted by one industry observer “...a human in the loop 
is essential: we are, unlike machines, able to take into account context and use general 
knowledge to put Al-drawn conclusions into perspective.” !° 


2. Drivers 


A variety of factors that have contributed to the growing use of FinTech generally have also 
spurred adoption of AI and machine learning in financial services.!7 On the supply side, 
financial market participants have benefitted from the availability of AI and machine learning 





See Slaughter and May and ASI Data Science (2017), “Superhuman Resources: Responsible deployment of AI in business,” 
Joint White Paper, June. 


16 See Finextra and Intel (2017), “The Next Big Wave: How Financial Institutions Can Stay Ahead of the AI Revolution,” 
May. 


See FSB FinTech Issues Group (2017), p.10. AI and machine learning are also being adopted widely in sectors such as 
health care, manufacturing, marketing, and many other areas. 


tools developed for applications in other fields. These include availability of computing power 
owing to faster processor speeds, lower hardware costs, and better access to computing power 
via cloud services.'® Similarly, there is cheaper storage, parsing, and analysis of data through 
the availability of targeted databases, software, and algorithms. There has also been a rapid 
growth of datasets for learning and prediction owing to increased digitisation and the adoption 
of web-based services.'? The declining cost of data storage and estimates of the volume of 
global data sets are shown in figure 3. 


Figure 3: Costs of storage and global data availability, 2009-2017 


an 


2010 2011 2012 2013 2014 2015 2016 2017 
= Global data created in zettabytes === Cost of 1 terabyte of storage, in $ (rhs) 


Source: Reinsel, Gantz and Rydning (2017); Klein (2017). One zettabyte is equal to one billion terabytes. 


The same tools driving advances in machine learning in search engines and self-driving cars, 
can be adopted in the financial sector. For example, entity recognition tools that enable search 
engines to understand when a user is referring to Ford Motor Company, rather than fording a 
river, are now used to quickly identify news or social media chatter relevant to publicly traded 
firms. As more firms adopt these tools, the financial incentives to access new or additional data 
and to develop faster and more accurate AI and machine learning tools may increase. In turn, 
such adoption and development of tools may affect incentives for yet other firms. 


A variety of technological developments in the financial sector have contributed to the creation 
of infrastructure and data sets. The proliferation of electronic trading platforms has been 
accompanied by an increase in the availability of high quality market data in structured 
formats.”° In some countries, such as the United States, market regulators allow publicly traded 
firms to use social media for public announcements. In addition to making digitised financial 





See IOSCO (2017), “Research Report on Financial Technologies (Fintech),” February, p. 6 on growth of computing power. 





Institute of International Finance (2017), “Deploying RegTech Against Financial Crime,” Report of the HF RegTech 
Working Group, March; David Reinsel, John Gantz and John Rydning (2017), “Data Age 2025: The Evolution of Data to 
Life-Critical,’ IDC White Paper, April; Andy Klein (2017), “Hard Drive Cost Per Gigabyte,” Backblaze, July. 


20 See IOSCO (2017), p. 41 for description of APIs and FIX protocols to assist in price discovery and market liquidity in 


corporate bond markets. 


data available for machine learning, the computerisation of markets has made it possible for AI 
algorithms to interact directly with markets, putting in real-time complex buy and sell orders 
based on sophisticated decision-making, in many cases with minimal human intervention. 
Meanwhile, retail credit scoring systems have become more common since the 1980s,”! and 
news has become machine readable since the 1990s. With the growth of data in financial 
markets as well as datasets — such as online search trends, viewership patterns and social media 
that contain financial information about markets and consumers — there are even more data 
sources that can be explored and mined in the financial sector. 


On the demand side, financial institutions have incentives to use AI and machine learning for 
business needs. Opportunities for cost reduction, risk management gains, and productivity 
improvements have encouraged adoption, as they all can contribute to greater profitability. In 
a recent study, industry sources described priorities for using AI and machine learning as 
follows: optimising processes on behalf of clients; working to create interactions between 
systems and staff applying AI to enhance decision-making; and developing new products and 
services to offer to clients.” In many cases these factors may also drive ‘arms races’ in which 
market participants increasingly find it necessary to keep up with their competitors’ adoption 
of AI and machine learning, including for reputational reasons (hype). 


Figure 4: Supply and demand factors of financial adoption of AI and machine learning 
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There is also demand due to regulatory compliance.”? New regulations have increased the need 
for efficient regulatory compliance, which has pushed banks to automate? and adopt new 
analytical tools that can include use of AI and machine learning. Financial institutions are 
seeking cost effective means of complying with regulatory requirements, such as prudential 


21 Examples include the FICO score in the US and Canada, Schufa scoring in Germany and firm-specific scores in Japan. 
22 Finextra and Intel (2017). 


23 From the perspective of technology companies offering AI and machine learning solutions for regulatory compliance by 
financial institutions, these factors can be considered supply-side. 


24 Tt is important to distinguish between ‘automation’ or ‘intelligent automation’ and use of AI and machine learning. This is 


a distinction that goes back to the use of algorithms to spread out orders in the 1990s. Most of those rule-based algorithms 
are not considered machine learning algorithms by the financial community, in contrast to the types of algorithms used 
today that do ‘learn’ from financial data, news and other data sources. 
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regulations, data reporting, best execution of trades, and rules on anti-money laundering and 
combating the financing of terrorism (AML/CFT). Correspondingly, supervisory agencies are 
faced with responsibility for evaluating larger, more complex and faster-growing datasets, 
necessitating more powerful analytical tools to better monitor the financial sector. Figure 4 
shows how these supply and demand factors fit together. 


A number of developments could impact future adoption of a broad range of financial 
applications of AI and machine learning. These developments include continued growth in the 
number of data sources and the timeliness of access to data; growth in data repositories, data 
granularity, variety of data types; and efforts to enhance data quality. Continued improvement 
in hardware, as well as AI and machine learning software as a service, including open-source 
libraries, will also impact continued innovation. Development in hardware includes processing 
chips and quantum computing that enable faster and more powerful AI. These developments 
could enable cheaper and broader access to AI and machine learning tools that are increasingly 
powerful. They could make more sophisticated real-time insights possible on larger datasets, 
such as real-time databases of online user behaviour or internet-of-things (IoT) sensors located 
around the world. 


At the same time, sophisticated software services are becoming more widely available. Some 
of the software services are open source libraries made available in the past few years that 
provide researchers with off-the-shelf-tools for machine learning. There are also a growing 
variety of vendors that provide machine learning for financial market participants, including 
some firms that scrape news and/or metadata and enable users to identify the specific features 
(webpages viewed, etc.) that correlate with the events they are interested in predicting. As 
services emerge to provide, clean, organise, and analyse these data for financial insights, the 
cost to users of incorporating sophisticated insights may fall significantly. Thus, at the same 
time, risks related to multiple users of the same information and techniques across the financial 
sector could grow (see section 4). 


The legal framework for relevant data will likely also impact the adoption of AI and machine 
learning tools. Breaches of personal data or uses of data that are not in the interests of consumers 
may be expected to lead to added data protection legislation (see annex A). In addition, the 
development of new data standards, new data reporting requirements, or other institutional 
changes in financial services can impact the adoption of AI and machine learning in specific 
markets. 


3. Selected use cases 


Financial stability implications depend critically on the uses of AI and machine learning. To 
assess these implications, questions to be considered would include which AI and machine 
learning tools are being used to make which types of decisions, on what time scales, to address 
which financial functions, and where and at what level human involvement is being integrated. 


AI and machine learning are being adopted for a number of purposes across the financial 
system. Examples include: 


° Sentiment indicators: Social media data analytics companies use AI and machine 
learning techniques to provide ‘sentiment indicators’ to a number of financial services 


10 


players. Investor sentiment indicators are being developed and sold to banks, hedge 
funds, high-frequency trading traders, and social trading and investment platforms.” 


° Trading signals: Machine learning can help firms to increase productivity and to 
reduce costs by quickly scanning and making decisions based on more sources of 
information than a human can (see section 3.3). Therein also lies a limitation of 
machine learning technology: by identifying and relying on patterns that were 
predictive of outcomes in the past, these tools are susceptible to false information.” 
For example, there were market moves across equities, bonds, foreign exchange, and 
commodities in April 2013 after trading algorithms reacted to a fraudulent news Tweet 
announcing two explosions at the White House. These types of issues may be 
exacerbated with more widespread use of machine learning. ”’ 


° AML/CFT and fraud detection: Seeking to increase productivity and simultaneously 
reduce costs and risks, while complying with regulations, some firms use AI for 
AML/CFT and fraud detection at financial institutions.’ Firms can also use machine 
learning for credit monitoring and risk mitigation purposes, (see section 3.4). 


This section considers four sets of use cases of AI and machine learning. These are: (i) 
customer-focused (or ‘front-office’) uses, including credit scoring, insurance, and client-facing 
chatbots; (ii) operations-focused (or ‘back-office’) uses, including capital optimisation, model 
risk management and market impact analysis; (iii) trading and portfolio management in 
financial markets; and (iv) uses of AI and machine learning by financial institutions for 
regulatory compliance (‘RegTech’) or by public authorities for supervision (‘SupTech’). For 
each of the use cases, a few examples of active or potential use cases are given, alongside 
estimates (where available) of the current adoption of technologies. The implications at the 
micro and macro level are reserved for sections 4 and 5. 


3.1 Customer-focused uses: credit scoring, insurance and client-facing chatbots 


Al and machine learning are already being applied in the front office of financial institutions. 
Large-scale client data are fed into new algorithms to assess credit quality and thus to price loan 
contracts. Similarly, such data can help assess risks for selling and pricing insurance policies. 
Finally, client interactions may increasingly be carried out by AI interfaces with so-called 
‘chatbots,’ or virtual assistance programs that interact with users in natural language. This 
section considers each in turn. 





25 See IOSCO (2017), p. 28. Social trading refers to a range of trading platforms that allow users to compare trading strategies 


or copy the trading strategy of other investors (see glossary). 


26 Of course, humans are also prone to error and manipulation. Machine learning algorithms may be superior to humans in 


detecting patterns that have not occurred before. At the same time, the use of such algorithms for automation could allow 
for a quicker and more large-scale dispersion of effects than for functions performed by individual human users. 

27 Tero Karpp, and Kate Crawford (2015), “Social Media, Financial Algorithms and the Hack Crash,” Theory Culture & 
Society 33(1): 73-92. 

28 Bart van Liebergen (2017), “Machine Learning: A Revolution in Risk Management and Compliance?” The Capco Institute 

Journal of Financial Transformation, April. 
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3.1.1 Credit scoring applications 


Credit scoring tools that use machine learning are designed to speed up lending decisions, while 
potentially limiting incremental risk. Lenders have long relied on credit scores to make lending 
decisions for firms and retail clients. Data on transaction and payment history from financial 
institutions historically served as the foundation of most credit scoring models. These models 
use tools such as regression, decision trees, and statistical analysis to generate a credit score 
using limited amounts of structured data. However, banks and other lenders are increasingly 
turning to additional, unstructured and semi-structured data sources, including social media 
activity, mobile phone use and text message activity, to capture a more nuanced view of 
creditworthiness, and improve the rating accuracy of loans. Applying machine learning 
algorithms to this constellation of new data has enabled assessment of qualitative factors such 
as consumption behaviour and willingness to pay. The ability to leverage additional data on 
such measures allows for greater, faster, and cheaper segmentation of borrower quality and 
ultimately leads to a quicker credit decision.” However, the use of personal data raises other 
policy issues, including those related to data privacy and data protections.” 


In addition to facilitating a potentially more precise, segmented assessment of creditworthiness, 
the use of machine learning algorithms in credit scoring may help enable greater access to 
credit. In traditional credit scoring models used in some markets, a potential borrower must 
have a sufficient amount of historical credit information available to be considered ‘scorable.’ 
In the absence of this information, a credit score cannot be generated, and a potentially 
creditworthy borrower is often unable to obtain credit and build a credit history. With the use 
of alternative data sources and the application of machine learning algorithms to help develop 
an assessment of ability and willingness to repay, lenders may be able to arrive at credit 
decisions that previously would have been impossible.*! While this trend may benefit 
economies with shallow credit markets, it could lead to non-sustainable increases in credit 
outstanding in countries with deep credit markets.** More generally, it has not yet been proved 
that machine learning-based credit scoring models outperform traditional ones for assessing 
creditworthiness. 


Over the past several years, a host of FinTech start-up companies targeting customers not 
traditionally served by banks have emerged. In addition to more commonly known online 
lenders that lend in the United States, one firm is using an algorithmic approach to data analysis 
and has expanded to overseas markets, particularly China, where the majority of borrowers do 
not have credit scores. Another firm, based in London, is working to provide credit scores for 
individuals with ‘thin’ credit files, using its algorithms and alternative data sources to review 
loan applications rejected by lenders for potential errors. Additionally, some companies are 





2 Stefan Lessmann, Bart Baesens, Hsin-Vonn Seow, and Lyn Thomas (2015), “Benchmarking state-of-the art classification 


algorithms for credit scoring: An update of research,” European Journal of Operational Research 247(1): 124-136. 
30 See CGFS and FSB (2017), p. 26. 


31 As an example, high-frequency online data on payments transactions can help to assess the creditworthiness of individuals 


and small businesses. 


32 For empirical evidence on the differential impact of credit inclusion on financial stability, see Ratna Sahay, Martin Čihák, 


Papa N’ Diaye, Adolfo Barajas, Srobona Mitra, Annette Kyobe, Yen Nian Mooi, and Seyed Reza Yousefi (2015), “Financial 
Inclusion: Can It Meet Multiple Macroeconomic Goals?” IMF Staff Discussion Note 15/17. One conclusion is that in 
countries with deep credit markets, greater credit-based financial inclusion can be associated with new vulnerabilities. 
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drawing on the vast amounts of data housed at traditional banks to integrate mobile banking 
apps with bank data and AI to assist with financial management and make financial projections, 
which may be first steps to developing a credit history. 


There are a number of advantages and disadvantages to using AI in credit scoring models. AI 
allows massive amounts of data to be analysed very quickly. As a result, it could yield credit 
scoring policies that can handle a broader range of credit inputs, lowering the cost of assessing 
credit risks for certain individuals, and increasing the number of individuals for whom firms 
can measure credit risk. An example of the application of big data to credit scoring could include 
the assessment of non-credit bill payments, such as the timely payment of cell phone and other 
utility bills, in combination with other data. Additionally, people without a credit history or 
credit score may be able to get a loan or a credit card due to AI, where a lack of credit history 
has traditionally been a constraining factor as alternative indicators of the likelihood to repay 
have been lacking in conventional credit scoring models. 


However, the use of complex algorithms could result in a lack of transparency to consumers. 
This ‘black box’ aspect of machine learning algorithms may in turn raise concerns. When using 
machine learning to assign credit scores make credit decisions, it is generally more difficult to 
provide consumers, auditors, and supervisors with an explanation of a credit score and resulting 
credit decision if challenged. Additionally, some argue that the use of new alternative data 
sources, such as online behaviour or non-traditional financial information, could introduce bias 
into the credit decision.** Specifically, consumer advocacy groups point out that machine 
learning tools can yield combinations of borrower characteristics that simply predict race or 
gender, factors that fair lending laws prohibit considering in many jurisdictions (see annex B). 
These algorithms might rate a borrower from an ethnic minority at higher risk of default because 
similar borrowers have traditionally been given less favourable loan conditions. 


The availability of historical data across a range of borrowers and loan products is key to the 
performance of these tools. Likewise, the availability, quality, and reliability of data on 
borrower-product performance across a wide range of financial circumstances is also key to the 
performance of these risk models. Also, the lack of data on new AI and machine learning 
models, and the lack of information about the performance of these models in a variety of 
financial cycles, has been noted by some authorities. *4 


3.1.2 Use for pricing, marketing and managing insurance policies 


The insurance industry is using machine learning to analyse complex data to lower costs and 
improve profitability. Since analysing data to drive pricing forms the core of insurance business, 
insurance-related technology, sometimes called ‘InsurTech,’ often relies on analysis of big data. 
Adoption of AI and machine learning applications in InsurTech is particularly high in the 
United States, UK, Germany and China.’ Many applications involve improvements to the 


33 For a thorough treatment, see Cathy O’Neil (2016), Weapons of Math Destruction: How Big Data Increases 
Inequality and Threatens Democracy, London: Allen Lane. 


34 See U.S. Treasury Department (2016), “Opportunities and Challenges in Online Marketplace Lending,” May; CGFS and 
FSB (2017), “FinTech Credit: Market Structure, Business Models and Financial Stability Implications,” May. 


Jim Struntz (2017), “AI on the insurance frontline,” Accenture Insurance Blog, July. 
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underwriting process, assisting agents in sorting through vast data sets that insurance companies 
have collected to identify cases that pose higher risk, potentially reducing claims and improving 
profitability.’ Some insurance companies are actively using machine learning to improve the 
pricing or marketing of insurance products by incorporating real-time, highly granular data, 
such as online shopping behaviour or telemetrics (sensors in connected devices, such as car 
odometers). Firms usually have access to those data through partnerships, acquisitions, or non- 
insurance activities. In many cases, firms need to ask for an active consent of the user whenever 
data protection regulation asks them to. 


AI and machine learning applications can substantially augment some insurance sector 
functions, such as underwriting and claims processing. In underwriting, large commercial 
underwriting and life or disability underwriting can be augmented by AI systems based on NLP. 
These applications can learn from training sets of past claims to highlight key considerations 
for human decision-makers. Machine learning techniques can be used to determine repair costs 
and automatically categorise the severity of vehicle accident damage.” In addition, AI may 
help reduce claims processing times and operational costs.*® Insurance companies are also 
exploring how AI and machine learning and remote sensors (connected through the “internet of 
things’) can detect, and in some cases prevent, insurable incidents before they occur, such as 
chemical spills or car accidents. 


It seems likely that these methods will achieve greater adoption. According to private sector 
estimates, global InsurTech investment totalled $1.7 billion in 2016. At the same time, 26 per 
cent of insurers provide monetary or non-monetary support (for example, coaching) to digital 
start-ups, and 17 per cent of insurers have an in-house venture capital fund or investment vehicle 
targeting technology.’ While the use of machine learning has the potential to produce more 
accurate pricing and risk assessment for insurance companies, there may be consumer 
protection concerns that stem from potential data errors or the exclusion of some groups (see 
section 5). 


3.1.3  Client-facing chatbots 


Chatbots are virtual assistants that help customers transact or solve problems. These automated 
programmes use NLP to interact with clients in natural language (by text or voice), and use 
machine learning algorithms to improve over time. Chatbots are being introduced by a range of 
financial services firms, often in their mobile apps or social media. While many are still in the 
trial phase, there is potential for growth as chatbots gain increasing usage, especially among the 
younger generations, and become more sophisticated. The current generation of chatbots in use 
by financial services firms is simple, generally providing balance information or alerts to 





See International Association of Insurance Supervisors, “FinTech Developments in the Insurance Industry,” 21 February 
2017. 


37 PWC (2016), “Top Issues: AI in Insurance: Hype or reality?” March. 





IAIS (2017), “Report on FinTech Developments in the Insurance Industry,” February. 


39 Accenture, “The Rise of Insurtech,” 2017. 


customers, or answering simple questions. It is worth observing that the increasing usage of 
chatbots is correlated with the increased usage of messaging applications. *° 


Chatbots are increasingly moving toward giving advice and prompting customers to act. In 
addition to assisting customers of financial institutions in making financial decisions, financial 
institutions can benefit by gaining information about their customers based on interactions with 
chatbots. While outdated infrastructure for client data storage has slowed the development of 
chatbots in financial institutions in many jurisdictions, Asian financial institutions and 
regulators have developed more sophisticated chatbots that are currently in active use. The 
insurance industry has also explored the use of chatbots to provide real-time insurance advice. 


3.2 Operations-focused uses 


Financial institutions can use AI and machine learning tools for a number of operational (or 
back-office) applications. Some of these applications include: (1) capital optimisation by banks; 
(ii) model risk management (back-testing and model validation); and (iii) market impact 
analysis (modelling of trading out of big positions). This section considers each of these in turn. 


3.2.1 Capital optimisation use case 


Capital optimisation, or the maximisation of profits given scarce capital, is a traditional function 
in running a bank that heavily relies on mathematical approaches. AI and machine learning 
tools build on the foundations of computing capabilities, big data, and mathematical concepts 
of optimisation to increase the efficiency, accuracy, and speed of capital optimisation. 
Optimisation of bank’s regulatory capital with machine learning has been a topic of interest 
from both academic and business professionals over the last several years. In 2012, a private 
sector observer noted that most banks said they had conducted meaningful programmes to 
optimise risk-weighted assets (RWA) and had seen 5 to 15 per cent RWA savings.*! Capital 
optimisation is also being done in the area of derivatives margin optimisation such as margin 
valuation adjustment (MVA).** New regulations around clearing and bilateral margining have 
increased the demand for sophisticated techniques for optimising capital and initial margin.” 


AI and machine learning could assist banks in optimising MVA, and recent research suggests 
that work is being done in this area.“ In the context of MVA optimisation, machine learning 
tries to reduce the initial margin for derivatives by a combination of: (a) executing pairs of 


40 According to one estimate, usage of the top 4 messaging services now surpasses the top 4 social network apps. See BI 


Intelligence (2016), “Messaging apps are now bigger than social networks,” September. 


41 McKinsey (2012), “Capital management, Banking’s new imperative,’ McKinsey Working papers on Risk, Number 38, 


November 2012. 
Margin valuation adjustment tries to determine the funding cost of the initial margin posted for a derivatives transaction. 


43 Kondratyev A, and G. Giorgidze, (2017), “MVA Optimisation with Machine Learning Algorithms, 23,” Social Science 
Research Network, January 2017. 


44M Heusser and P Varhol, (2016) “An Intro to Genetic Algorithms,” InfoWorld, December 2016; Qinghai Bai (2010), 
“Analysis of Particle Swarm Optimisation Algorithm,” Computer and Information Science 3(1), February. The first paper 
discusses genetic algorithms, which are a method that optimises an output or outputs with successive derived equations 
based on predetermined hypotheses. The second discusses particle swarm optimisation, which is a method and algorithm 
based on swarm intelligence, derived from research on the movement behaviour of flocks of birds and schools of fish. Both 
offer machine learning techniques to optimise funding costs in the constraints of the leverage ratio-implied capital charge. 
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offsetting derivative trades; (b) executing offsetting strategies with the same dealer; (c) novating 
trades from one dealer portfolio to another. Machine learning finds the best combination of the 
initial margin reducing trades at a given time based on the degree of initial margin reduction in 
the past under different combinations of those trades. A likely implication of these advances in 
RWA and MVA optimisation is a reduction in the traditionally calibrated regulatory capital and 
larger reliance on the non-optimisable capital regulatory tools. 


3.2.2 Model risk management (back-testing and model validation) and stress testing 


Academics and practitioners often cite back-testing and model validation as areas where 
progress with AI and machine learning will likely be soon visible.“ Banks are considering 
machine learning to make sense of large, unstructured and semi-structured datasets and to police 
the outputs of primary models. Back-testing is important because it is traditionally used to 
evaluate how well banks’ risk models are performing. In the last years, US and European 
prudential regulators focused on back-testing and validation used by banks by providing 
guidance on model risk management.*° Using a range of financial settings for back-testing 
allows for consideration of shifts in market behaviour and other trends, hopefully reducing the 
potential for underestimating risk in such scenarios.*” 


Some applications are already live. For instance, one global corporate and investment bank is 
using unsupervised learning algorithms in model validation. Its equity derivatives business has 
used this type of machine learning to detect anomalous projections generated by its stress- 
testing models. Each night, these models produce over three million computations to inform 
regulatory, internal capital allocations and limit monitoring. A small fraction of these 
computations are extreme, and knocked out of the normal distribution of results by a quirk of 
the computation cycle or faulty data inputs. Unsupervised learning algorithms help model 
validators in the ongoing monitoring of internal and regulatory stress-testing models, as they 
can help determine whether those models are performing within acceptable tolerances or 
drifting from their original purpose. They can also provide additional input to operational risk 
models, such as the vulnerability of organisations to cyber-attacks. 


Similarly, AI and machine learning techniques are also being applied to stress testing. The 
increased use of stress testing following the financial crisis has posed challenges for banks as 
they work to analyse large amounts of data for regulatory stress tests. One provider of AI and 
machine learning tools has worked closely with a large financial institution to develop tools to 
assist them in modelling their capital markets business for bank stress testing. The tools 
developed aim to limit the number of variables used in scenario analysis for the loss given 
default and probability of default models. By using unsupervised learning methods to review 





Louie Woodall (2017), “Model risk managers eye benefits of machine learning,” Risk, April. 





For example, see Federal Reserve Board (2011), “Supervisory guidance on model risk management,” SR Letter 11-7, April; 
FDIC (2017), “Supervisory guidance on model risk management,” FIL-22-2017, June; ECB (2017), “Guide to the Targeted 
Review of Internal Models,” February. 














Model validation is defined in industry guidance as “the set of processes and activities intended to verify that models are 
performing as expected, in line with their design objectives, and business uses [to identify] potential limitations and 
assumptions, and assesses their possible impact.” See Clayton Mitchell (2016), “Model Validation: For Elements of 
Determining the Accuracy of Your Model,” British Bankers Association, January. 
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large amounts of data, the tools can document any bias associated with selection of variables, 
thereby leading to better models with greater transparency. 


3.2.3 Market impact analysis (modelling of trading out of big positions) 


Aland machine learning can complement conventional market impact models.** Firms can use 
AI to obtain more information from sparse historical models, or help identify non-linear 
relationships in order flow. Machine learning can be used to create ‘trading robots’ that then 
teach themselves how to react to market changes. Market impact analysis involves evaluating 
the effect of a firm’s own trading on market prices. Since firms are concerned about the impact 
of trades, especially of large trades, on market prices, more accurate estimation of this impact 
is key to timing trades and minimising trading execution costs. 


Firms are investigating using AI tools to assess the market impact of a given trade. The effect 
of a firm’s own trading on market prices is notoriously hard to model, especially for less liquid 
securities, where data on comparable past trades are scarce. AI tools may help by augmenting 
models already in use, or by introducing a machine learning approach to minimise trading 
impact on prices and liquidity. For the most active systematic funds, as much as two-thirds of 
the gain on trades are estimated to be lost to market impact costs. AI tools may help by 
augmenting models already in use, or by introducing a machine learning approach to minimise 
trading impact on prices and liquidity for trading both into and out of large market positions, or 
as a part of every-day trading strategies. 


Machine learning is often used to identify groups of bonds that behave similarly to each other. *° 
By doing so, they can rely on many more data points, providing better estimates of price 
movements when the market is thin. The resulting tool groups bonds into broad, intuitively 
similar buckets and then, using cluster analysis, collects the most comparable products together 
in each bucket, to score the liquidity of individual bonds. 


Also, AI can be used to help identify how the timing of trades can minimise market impact. 
Market impact models can be developed that describe how the effect of a trade depends on 
previous trades as a starting point. The models attempt to avoid scheduling trades too closely 
together to avoid having a market impact greater than the sum of its parts. These models can be 
used to set out the best possible trading schedules for a range of scenarios and then tweak the 
schedule as the real trade progresses, using supervised learning techniques to make the short- 
term predictions determining those tweaks. Banks are also testing reinforcement learning to 
teach artificial intelligence tools to react to order imbalance and queue position in the limit 
order book. 


48 Sebastian Day (2017), “Quants turn to machine learning to model market impact,” RISK Magazine, April. 
49 Day (2017). 


50 For example, one firm uses machine learning to assess the liquidity of bonds. Every bond is quantitatively measured against 
a range of common features such as currency, duration, time to maturity and amount outstanding. Those measurements 
determine its position within a theoretical multi-dimensional space. For example, trading 500 lots of an obscure US 
Treasury bond, the tool will identify other US Treasury bonds the shortest distance away within that space. The tool will 
then use their combined pool of data to calibrate the parametric model. 
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3.3 Trading and portfolio management 


AI and machine learning techniques are active areas of research and development for asset 
managers and trading firms. In addition to significant research and development (R&D), some 
firms now use machine learning to devise trading and investment strategies. The extent to which 
AI investment strategies are autonomous or incorporate human oversight varies on a case-by- 
case basis. In this section, we distinguish between trading execution (primarily sell-side) and 
portfolio management (buy-side). 


3.3.1 AI and machine learning in trading execution 


Trading firms are looking to AI and machine learning to use data to improve their ability to sell 
to clients. For example, analysing past trading behaviour can help anticipate a client’s next 
order. Trading generates large quantities of data, and this scale is typically required by machine 
learning tools to work effectively. If the current trend to increasing use of voice-to-text services 
continues, this will generate additional data from trades executed over the phone, which can be 
integrated with the pool of data from electronic platforms. 


Al and machine learning can more pro-actively manage risk exposures. Machine learning can 
serve as a basis for risk modelling by exchanges to determine when members’ trading account 
positions may have increased risk profiles that may warrant intervention. For large trading firms 
such as banks, the use of a central risk trading book, or risk management techniques based on 
big data analysis, have enabled these firms to manage risks and optimise their use of capital by 
centralising the risks that arise from various parts of their businesses. 


AI and machine learning can help compliance with trading regulations. A RegTech application 
of AI to trading is voice-to-text technology powered by deep learning. This helps firms meet 
pre-trade and post-trade transparency requirements for non-equity markets.*! 


33.2 Scope for the use of AI and machine learning in portfolio management 


In portfolio management, AI and machine learning tools are being used to identify new signals 
on price movements and to make more effective use of the vast amount of available data and 
market research than with current models. Machine learning tools work on the same principles 
as existing analytical techniques used in systematic investing. The key task is to identify signals 
from data on which predictions relating to price level or volatility can be made, over various 
time horizons, to generate higher and uncorrelated returns. 


Among asset managers, machine learning is used most extensively by systematic (‘quant’) 
funds, most of which are hedge funds.*” An AI unit tends to sit within a larger team at an asset 
manager to aid with portfolio construction. One view in the industry is that for AI and machine 
learning to be effective, both traders and quants need to have good oversight and understanding 
of the tools used. Many quant funds state that they are not comfortable fully automating and 


Under the Markets in Financial Instruments Directive (MiFID II), for instance, dealers designated as ‘systematic 
internalisers’ will be unable to quote prices to a client for some bilateral trades unless they also simultaneously broadcast 
the same quote to the wider market — a task that is realisable if the quote is provided on a screen, but more difficult via a 
phone. 


While these players have been using quantitative techniques since long before the recent advances in deep learning, they 
are widely regarded as some of the most avid adopters of these techniques. 
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implementing a model if they cannot understand how a particular prediction is made. The 
concentration of machine learning tools among quant funds reflects how machine learning is 
fundamentally an approach to generating predictive power from data, which distinguishes it 
from investment approaches that use greater discretion and judgment. 


At the moment, machine learning likely only drives a minor subset of quant funds’ trades. Quant 
funds manage on the order of $1 trillion in assets, out of total assets under management (AUM) 
invested in mutual funds globally in excess of $40 trillion. The market share of quant funds has 
not changed drastically in the years since the crisis, but between 2013 and 2016 the proportion 
of trades carried out by quant funds, on one measure, approximately doubled from 13% to 
27%.* In turn, some portion of the trading is based on machine learning. It is hard to quantify 
precisely what proportion use machine learning for several reasons: 


e Firms are hesitant to share proprietary information. 


e When firms share information on their use of machine learning, there is not always a 
standard definition or understanding of what is included within the concept. 


e Some investments or trades may be made on a discretionary basis but informed (to 
varying extents) by the use of machine learning." 


One contact in the industry estimates that ‘pure’ AI and machine learning players have about 
$10 billion in assets under management, but that this figure is growing rapidly.°° 


In addition to the use by fund managers, specialist firms are making available to asset managers 
machine learning tools to gain insight from the vast volume of news and market research 
available. In other cases, asset managers are themselves building indicators, using AI capability 
supplied by third parties. One general issue is that useful trading signals derived from AI and 
machine learning strategies may follow a decay function over time, as data are more widely 
used and hence become less valuable for gaining an edge over other investors. 56 


3.4 AI and machine learning in regulatory compliance and supervision 


AI and machine learning techniques are being used by regulated institutions for regulatory 
compliance, and by authorities for supervision. RegTech is often regarded as the subset of 
FinTech that focus on facilitating regulatory compliance more efficiently and effectively than 
existing capabilities.°’ The total RegTech market is expected to reach $6.45 billion by 2020, 


53 Tabb Group, referenced in Gregory Zuckerman and Bradley Hope (2017), “The Quants Run Wall Street Now,” Wall Street 
Journal, 22 May. 


54 For some quant funds, machine learning tools inform investment strategies that are implemented by a person. Other firms 


provide information generated by machine learning to asset managers. For example, one firm’s machine learning engine 
shows how asset prices have behaved historically in response to market events. In other cases, it appears that firms are 
using machine learning systems to manage portfolios and to execute trades automatically. One firm runs an automated fund 
using an evolutionary computation approach, using a large network of central processing units to randomly generate 
trillions of trading “genes;” from which the system selects and “breeds” the best-performing 0.01%. 


55 This is based on bilateral discussions by the FSB secretariat with an investor focused on AI and machine learning funds. 


56 See for example Luke Smolinski (2017), “Wolfe aims to shake up research with AI push,” Risk.net, 23 May. 


5 FCA (2015), “Call for Input: Supporting the development and adoption of RegTech,” November; Institute of International 


Finance (2015), “Regtech: exploring solutions for regulatory challenges,” October. 
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growing at a compound annual growth rate (CAGR) of 76%.58 SupTech is the use of these 
technologies by public sector regulators and supervisors. Within SupTech, the objective of AI 
and machine learning applications is to enhance efficiency and effectiveness of supervision and 
surveillance. While there can be overlap in the terms,*? the two applications are discussed here 
separately. Some of the examples below are from the academic community. While not yet being 
applied by regulatory or supervisory bodies, they represent potential applications in this sector. 
The use cases are grouped by the function for which they are used, namely regulatory 
compliance; regulatory reporting and data quality; monetary policy and systemic risk analysis; 
and surveillance and fraud detection. 


3.4.1 RegTech: applications by financial institutions for regulatory compliance 


For analysing unstructured data, RegTech can use machine learning combined with NLP. 
Besides being applied to the monitoring of behaviour and communication of traders for 
transparency and market conduct, machine learning together with NLP can interpret data inputs 
such as e-mails, spoken word, instant messaging, documents, and metadata. This in turn begs 
the issue of the boundaries for the employee surveillance policy. Some regulated institutions 
are experimenting with cases seeking to enhance their ability to comply with product suitability 
requirements. 


NLP could be used by asset management firms to cope with new regulations. In the EU, 
investment managers have to comply with specific requirements in the Markets in Financial 
Instruments Directive (MiFID II), the Undertakings for Collective Investments in Transferrable 
Securities (UCITS) Directive, and the Alternative Investment Fund Managers Directive 
(AIFMD). Firms could potentially leverage NLP and other machine learning tools to interpret 
these regulations into a common language. They could then analyse and codify the rules for 
automation into the integrated risk and reporting systems to help firms comply with the 
regulations. This could bring down the cost, effort and time needed to interpret and implement 
new and updated regulations for fund managers. 


Knowing the identity of customers (‘know your customer’ or KYC) is another area where AI 
and machine learning are applied to address one of the biggest pain points in the financial 
industry, both with regards to user experience and regulator expectations. The KYC process is 
often costly, laborious, and highly duplicative across many services and institutions. Machine 
learning is increasingly used in remote KYC of financial services firms to perform identity and 
background pre-checks. It is predominantly used in two ways: (1) evaluating whether images 
in identifying documents match one another, and (2) calculating risk scores on which firms 
determine which individuals or applications need to receive additional scrutiny. Machine 
learning-based risk scores are also used in ongoing periodic checks based on public and other 
data sources, such as police registers of offenders and social media services. Use of these 
sources may enable risk and trust to be assessed quickly and often cheaply. Firms can use risk 
scores on the probability of customers raising “red flags” on KYC checks to help make 
decisions on whether to proceed with the time and expense of a full background check. 





58 Frost & Sullivan (2017), “Global Forecast of RegTech in Financial Services to 2020,” March. 


For instance, some authors use RegTech to refer to applications used by regulators, as well. See Douglas W. Arner, Janos 
Barberis and Ross P. Buckley (2017), “FinTech, RegTech and the Reconceptualization of Financial Regulation,” 
Northwestern Journal of International Law and Business 37(3): 371-413. 
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Nonetheless, concerns about their accuracy have kept some financial services from 
incorporating these tools. 


3.4.2 Uses for macroprudential surveillance and data quality assurance 


AI and machine learning methods may help to improve macroprudential surveillance by 
automating macroprudential analysis and data quality assurance. A series of new reporting 
requirements across jurisdictions has led to a greater volume and frequency of reported data, as 
well as greater resources required from financial institutions to complete reporting on time. In 
some cases (for example, transactions data in MiFID, AIFMD templates, etc.), the volume of 
data received can be challenging for the authorities receiving the data, such that it cannot be 
used to its full potential using traditional methods. Moreover, substantial errors, blank fields, 
and other data quality issues are often more prevalent in new datasets, and additional checks 
and data quality assurance are needed. Machine learning can help improve data quality, for 
example, by automatically identifying anomalies (potential errors) to flag them to the 
statistician and/or the data-providing source. This may allow for both lower-cost and higher- 
quality reporting and more efficient and effective data processing and macroprudential 
surveillance of data by authorities.” 


Similarly, AI and machine learning could help trade repositories (TRs) tackle data quality 
issues, increasing the value of TR data to authorities and the public. Authorities report that 
overcoming data quality issues continues to be a key challenge to making full use of TR data.*! 
Application of machine learning techniques may help TRs — for over-the-counter (OTC) 
derivatives or (where applicable) other types of transactions, such as exchange-traded 
derivatives or securities financing transactions — improve data quality. Specifically, 
appropriately trained machine learning algorithms could help identify data gaps, data 
inconsistencies, and fat-finger errors, as well as match likely pairs of transactions and/or 
interpolate missing data. The same techniques can be used by authorities, themselves. In this 
context, the Autorité des marchés financiers du Québec reports that it has successfully tested in 
its FinTech Laboratory a supervised machine learning algorithm able to recognise distinct 
categories from unstructured free text fields in OTC derivatives data, such as the floating leg of 
swaps. Implementation of alerts based on this algorithm is underway to automatically detect 
transactions that are not compliant with mandatory clearing requirements. 


3.4.3 SupTech: uses and potential uses by central banks and prudential authorities 


Machine learning can be applied to systemic risk identification and risk propagation channels. 
Specifically, NLP tools may help authorities to detect, measure, predict, and anticipate, among 
other things, market volatility, liquidity risks, financial stress, housing prices, and 
unemployment.® In a recent Banca d’Italia (BdI) study, still in progress, textual sentiment 


In addition to the applications of AI and machine learning, there are a number of potential applications of distributed ledger 
technology (DLT), cloud computing and digital identity to regulatory reporting. These RegTech applications are beyond 
the scope of this paper. See IIF (2017) for more detail. 


6! See e.g. FSB (2017), Review of OTC derivatives market reforms: Effectiveness and broader effects of the reforms, at p. 28 
62 AMF (2017), “AMF creates Fintech lab and signs partnership with R3,” press release, April. 


& David Bholat, Stephen Hansen, Pedro Santos, and Cheryl Schonhardt-Bailey (2015), Text mining for central banks, Bank 
of England CCBS Handbook No. 33. 
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derived from Twitter posts is used as a proxy for the time-varying retail depositors’ trust in 
banks. The indicator is used to challenge the predictions of a banks’ retail funding model, and 
to try to capture possible threats to financial stability deriving from an increase of public distrust 
in the banking system. Furthermore, at the BdI, in order to extract the most relevant information 
available on the web, newspaper articles are processed through a suitable NLP pipeline that 
evaluates their sentiment. In another study, academics developed a model using computational 
linguistics and probabilistic approaches to uncover semantics of natural language in mandatory 
US bank disclosures. The model found risks as early as 2005 related to interest rates, mortgages, 
real estate, capital requirements, rating agencies and marketable securities.° Other studies are 
able to predict and anticipate market outcomes and economic conditions, including volatility® 
and growth. 


Use of machine learning combined with NLP can be used to identify patterns for further 
attention from supervisors in large and complex data. Machine learning can also be used with 
NLP to link trading databases to other information on market participants. This could include, 
for example, the ability to integrate and compare trading activity information with behavioural 
data like communications and to compare normal trading scenarios with those that may have 
substantial deviations, triggering the need for further analysis.°” 


Central banks can use AI to assist with monetary policy assessments. A 2015 survey of central 
banks’ use of and interest in big data reported, among other things, that central banks expected 
a growing use of big data for macroeconomic and financial stability purposes. The most 
prevalent expected use was for economic forecasting, in particular for economic indicators such 
as inflation and prices. For instance, 39% of central banks expect to ‘nowcast,’ or predict in real 
time, retail home prices using big data. AI can be used to forecast unemployment, GDP, 
industrial production, retail sales, tourism activity, and the business cycle (for example, with 
sentiment indicators and nowcasting techniques).°* © 


Recent research highlights how these methods could be used. Researchers at Columbia 
University have recently combined newly developed machine learning approaches with 
observational studies to enable public authorities and market participants to: (i) ‘score’ policy 
choices and link them to indicators of financial sector performance; (ii) simulate the impact of 
policies under varying economic and political conditions; and (iii) detect the rate of change of 





64 See Kathleen Weiss Hanley and Gerard Hoberg (2016), “Dynamic Interpretation of Emerging Systemic Risks,” working 


paper, October. 


65 See Harry Mamaysky and Paul Glasserman (2016), “Does Unusual News Forecast Market Stress?” Columbia Business 


School Research Paper No. 15-70, April. 


66 See Samuel Fraiberger (2016), “News Sentiment and Cross-Country Fluctuations,” February. 


67 See Bart van Liebergen, “Machine Learning: A Revolution in Risk Management and Compliance?” The CAPCO Institute 


Journal of Financial Transformation, 2017. See also, Singapore Monetary Authority development of algorithms to detect 
and identify trading accounts suspected of syndicated activity, Ravi Menon, Managing Director, MAS, “Financial 
Regulation — The Forward Agenda,” March 20, 2017. 


68 See Irving Fisher Committee on Central Bank Statistics (2015), “Central banks’ use of and interest in “big data,” October. 


© A growing body of existing research done by academics and others suggests that machine learning tools do in fact make it 


possible to better detect, measure, predict, anticipate, and even nowcast market outcomes. See the following for more 
examples: Cindy K. Soo (2013), “Quantifying Animal Spirits: News Media and Sentiment in the Housing Market,” 
University of Pennsylvania The Wharton School, January; Hal Varian and Hyunyoung Choi (2009), “Predicting the Present 
with Google Trends” Google Research Blog, April. 
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market innovation by comparing trends of policy efficacy over time.” With the aim of studying 
the redistributive effects of fiscal policy over different municipalities, a study from the BdI 
employs a dynamic factor model and utilises a dataset containing variables from different 
sectors of the economy. In order to select the statistically most relevant independent variables 
they use automatic regression variable selection.’' At the Office of Financial Research (OFR), 
researchers are evaluating the potential for machine learning tools to identify new financial 
innovations receiving more attention from market participants in financial publications. OFR 
researchers have also used machine learning to extract sentiment and key topics from financial 
publications in order to evaluate the relationship between news, attention, and financial 
stability. 


3.4.4 Uses by market regulators for surveillance and fraud detection 


Some regulators are using AI for fraud and AML/CFT detection. The Australian Securities and 
Investments Commission (ASIC) has been exploring the quality of results and potential use of 
NLP technology to identify and extract entities of interest from evidentiary documents. ASIC 
is using NLP and other technology to visualise and explore the extracted entities and their 
relationships. In order to fight criminal activities carried out through the banking system (such 
as money laundering), BdI collects detailed information on bank transfers and correlates this 
information with information from newspaper articles. The correlation involves both structured 
and unstructured data for file sizes of more than 50 gigabytes. In the same vein, the Monetary 
Authority of Singapore (MAS) is exploring the use of AI and machine learning in the analysis 
of suspicious transactions to identify those transactions that warrant further attention, allowing 
supervisors to focus their resources on higher risk transactions. Investigating suspicious 
transactions is time consuming and often suffers from a high rate of false positives, due to 
defensive filings by regulated entities. Machine learning is being used to identify complex 
patterns and highlight the suspicious transactions that are potentially more serious and warrant 
closer investigation. Coupled with machine learning methods to analyse the granular data from 
transactions, client profiles, and a variety of unstructured data, machine learning is being 
explored to uncover non-linear relationships among different attributes and entities, and to 
detect potentially complicated behaviour patterns of money laundering and the financing of 
terrorism not directly observable through suspicious transactions filings from individual 
entities. 


Market regulators can also use these techniques for disclosure and risk assessment. The US 
Securities and Exchange Commission (SEC) staff leverages “big data” to develop text analytics 
and machine learning algorithms to detect possible fraud and misconduct. Certain risk 
assessment tools are beginning to move into the AI space.” For instance, the SEC staff uses 





7 Sharyn O’Halloran, Sameer Maskey, Geraldine McAllister, David K. Park and Kaiping Chen (2015), “Big Data and the 
Regulation of Financial Markets,” IEEE/ACM International conference on Advances in Social Networks Analysis and 
Mining. 


Monica Andini, Emanuele Ciani, Guido de Blasio, Alessio D’Ignazio, and Viola Salvestrini (2017), “Targeting policy- 
compliers with Machine Learning: An application to a tax rebate program in Italy,” Banca d’Italia working paper, 
forthcoming. 


The SEC staff has noted that it extracts words and phrases from narrative disclosures in forms and filings and using human 
written rules to define patterns in document to systematically measure and assess how emerging growth companies are 
availing themselves of JOBS Act provisions. See Scott Bauguess (2017), “The Role of Big Data, Machine Learning and 
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machine learning to identify patterns in the text of SEC filings. With supervised learning, these 
patterns can be compared to past examination outcomes to find risks in investment manager 
filings. The SEC staff notes that these techniques are five times better than random at finding 
language that merits a referral to enforcement. While the results can generate false positives 
that can be explained by non-nefarious actions and intent, these nonetheless provide 
increasingly important signals to prioritise examination.’? For investment advisers, the SEC 
staff compiles structured and unstructured data. Unsupervised learning algorithms are used to 
identify unique or outlier reporting behaviours — including both topic modelling and tonality 
analysis. ’* The output from this first stage is then combined with past examination outcomes 
and fed into a second-stage, machine learning algorithm to predict the presence of idiosyncratic 
risks at each investment advisor.” In Australia, ASIC has also used machine learning software 
to identify misleading marketing in a particular sub-sector, such as unlicensed accountants in 
the provision of financial advice.” 


4. Micro-financial analysis 


From a micro-financial point of view, the application of AI and machine learning to financial 
services may have an important impact on financial markets, institutions and consumers.” In 
this section, potential changes to incentives and behaviour and how they may affect financial 
stability, for better or worse, are considered. 


4.1 Possible effects of AI and machine learning on financial markets 


Since AI and machine learning have the potential to substantially enhance the efficiency of 
information processing, thereby reducing information asymmetries, applications of AI and 
machine learning have the potential to strengthen the information function of the financial 
system. ’* The mechanisms whereby this improvement may occur include: ” 


a) AI and machine learning may enable certain market participants to collect and analyse 
information on a greater scale. In particular, these tools may help market participants to 
understand the relationship between the formulation of market prices and various factors, 


AI in Assessing Risks: a Regulatory Perspective,” Speech by Acting Director and Acting Chief Economist, Division of 
Economics and Risk Analysis, OpRisk North America, June 21. 


3 See Bauguess (2017). See also Gerard Hoberg and Craig M. Lewis (2015), “Do Fraudulent Firms Produce Abnormal 
Disclosure?” Working paper. 
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financial systems: Towards a synthesis of function and structure,” Journal of Investment Management 3(1): 1-23. 
79 The Bank of Japan held an AI conference on April 13, 2017, and the market views described in this chapter are in line with 


those expressed in this conference. 


24 


such as in sentiment analysis. This could reduce information asymmetries and thus 
contribute to the efficiency and stability of markets.*° 


b) Aland machine learning may lower market participants’ trading costs. Moreover, AI and 
machine learning may enable them to adjust their trading and investment strategies in 
accordance with a changing environment in a swift manner, thus improving price discovery 
and reducing overall transaction costs in the system. 


Nonetheless, if many market participants come to use similar AI and machine learning 
programmes in areas such as credit scoring or financial market activities, the consequent 
correlated risks may entail financial stability risks. If machine learning-based traders 
outperform others, this could in the future result in many more traders adopting similar machine 
learning strategies (even if this may also reduce the profitability of such strategies). While there 
is no evidence of this occurring to date, this could become relevant with greater adoption of 
such trading strategies. As with any herding behaviour in the market, this has the potential to 
amplify financial shocks. Moreover, advanced optimisation techniques and predictable patterns 
in the behaviour of automated trading strategies could be used by insiders or by cybercriminals 
to manipulate market prices.*! 


4.2 Possible effects of AI and machine learning on financial institutions 


AI and machine learning have the potential to enhance the efficiency and profitability of 
financial institutions, while reducing their costs and risks, through various channels. Greater 
profitability could aid the build-up of buffers and ultimately benefit system-wide stability: 


a) AI and machine learning may enhance machine-based processing of various operations in 
financial institutions, thus increasing revenues and reducing costs. For example, if AI and 
machine learning help to identify customers’ needs and better target or tailor products to 
profitable customers, financial institutions could more efficiently allocate resources toward 
serving those customers that account for substantial fees or have the potential for future 
growth. Automating routine business processes may allow for lower operating costs. 


b) AI and machine learning can be used for risk management through earlier and more accurate 
estimation of risks. For example, to the extent that AI and machine learning enable decision- 
making based on past correlations among prices of various assets, financial institutions could 
better manage these risks. Tools that mitigate tail risks could be especially beneficial for the 
overall system. Also, AI and machine learning could be used for anticipating and detecting 
fraud, suspicious transactions, default, and the risk of cyber-attacks, which could result in 
better risk management. But AI and machine learning based tools might also miss new types 
of risks and events because they could potentially ‘overtrain’ on past events. While AI and 
machine learning tools hold potential to improve risk management, the recent deployment 
of these strategies means that they remain untested at addressing risk under shifting financial 
conditions. 





80 See IOSCO (2017), p. 28. 
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c) The data intensity and open-source character of research in AI and machine learning may 
encourage collaboration between financial institutions and other industries, such as e- 
commerce and sharing economy businesses. 


Nonetheless, use of AI and machine learning risks creating “black boxes’ in decision-making 
that could create complicated issues, especially during tail events. In particular, it may be 
difficult for human users at financial institutions — and for regulators — to grasp how decisions, 
such as those for trading and investment, have been formulated. Moreover, the 
communication mechanism used by such tools may be incomprehensible to humans, thus 
posing monitoring challenges for the human operators of such solutions. ® If in doubt, users of 
such AI and machine learning tools may simultaneously pull their ‘kill switches,’ that is 
manually turn off systems. After such incidents, users may only turn systems on again if other 
users do so in a coordinated fashion across the market. This could thus add to existing risks of 
system-wide stress and the need for appropriate circuit-breakers. 


In addition, if AI and machine learning based decisions cause losses to financial intermediaries 
across the financial system, there may be a lack of clarity around responsibility.** For example, 
if a specific AI and machine learning application developed by a third party resulted in large 
losses, is the institution that conducted the trading solely responsible for the losses? Or would 
regulators or other parties be able to pursue potential claims against the application developer? 
Could more widespread use of AI and machine learning, including by non-traditional market 
players, impact the nature of supervision? Furthermore, there are open questions about 
(identifying) potential collusion among trading applications that rely on deep learning. 
Specifically, if algorithms interact in ways that would be considered collusion if done by human 
agents, then as with human agents, proof of intent may be an issue. In this light, there may be a 
number of legal uncertainties (see annex A). Finally, the lack of transparency around 
applications may be problematic for both institutions and regulators when it may not be possible 
to understand how undesired events occurred and when steps may need to be taken to prevent 
a recurrence. 


Any uncertainty in the governance structure in the use of AI and machine learning might 
increase the risks to financial institutions.*° If each investor makes their investment without 
fully understanding the applications and his or her possible losses in tail events, the aggregate 
risks could be underestimated. In addition, any uncertainty in the governance structure could 
substantially increase the costs for allocating losses, including the possible costs of litigation. 
In this regard, financial institutions applying AI and machine learning to their businesses need 
to establish well-designed governance and maintain auditability. 





For an article concisely describing the problems of black boxes in AI decision-making, see Will Knight (2017), “The Dark 
Secret at the Heart of AI,” MIT Technology Review, April. 
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Finally, there may be important third-party dependencies. In the development of AI and 
machine learning to date there is a high reliance on a relatively small number of third-party 
technological developers and service providers. This third-party reliance could be relevant for 
market participants and financial institutions in the future. For instance, if a major provider of 
AI and machine learning tools were to become insolvent or suffer an operational risk event, this 
could lead to operational disruptions at a large number of financial institutions at the same time. 
These risks may become more important in the future if AI and machine learning are used for 
‘mission-critical’ applications of financial institutions. 


4.3 Possible effects of AI and machine learning on consumers and investors 


If AI and machine learning reduce the costs and enhance the efficiency of financial services, 
consumers could obtain a number of benefits. 


a) Consumers and investors could enjoy lower fees and borrowing costs if AI and machine 
learning reduce the costs for various financial services. 


b) Consumers and investors could have wider access to financial services. For example, 
applications of AI for robo-advice might facilitate people’s use of various asset markets for 
their investments. Moreover, AI and machine learning, through advanced credit scoring for 
FinTech lending, might make wider sources of funds available to consumers and small and 
medium enterprises (SMEs). 


c) AI and machine learning could facilitate more ‘customised’ and ‘personalised’ financial 
services through big data analytics. For example, AI and machine learning might facilitate 
the analysis of big data, thus clarifying the characteristics of each consumer and/or investor 
and allowing firms to design well-targeted services. Nonetheless, the use of consumers’ data 
may entail issues of data privacy and information security.8° Moreover, since AI and 
machine learning analytics could analyse the characteristics of each customer through public 
data, it would be necessary to consider how the output of customer analysis should be 
protected, while protecting the anonymity of each consumer and facilitating the safe and 
efficient use of big data for better services. In addition, establishing well-designed 
governance structures for financial service providers using AI and machine learning would 
be important for consumer and investor protection purposes. 


Avoiding discrimination in credit scoring, credit provision, and insurance is also an important 
topic. Even where data on sensitive characteristics such as race, religion, gender, etc. are not 
collected, AI and machine learning algorithms may create outcomes that implicitly correlate 
with those indicators, for example, based on geography or other characteristics of individuals. 
There is ongoing research on how to address and mitigate these biases. This is a key area in the 
broader discussion on AI ethics (see annex B). 


86 Since AI and machine learning analytics could analyse the characteristics of each customer through public data, it would 
be necessary to consider how the output of customer analyses and protecting the anonymity of each consumer and 
facilitating the safe and efficient use of big data for better services. In addition, establishing well-designed governance 
structures for financial service providers using AI and machine learning would be important for consumer protection 
purposes. On issues of data privacy and information security, see Haruhiko Kuroda (2016), “Information Technology and 
Financial Services: The Central Bank’s perspective,” Remarks by Governor Kuroda at the FinTech Forum, August 23. 
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4.4 Current regulatory considerations regarding the use of AI and machine learning 


Because AI and machine learning applications are relatively new, there are no known dedicated 
international standards in this area. Yet in light of some of the potential risks identified above, 
a few efforts by international standards-setters and similar international fora of regulators 
deserve note. For example, several international standards-setters have considered risks 
associated with algorithmic trading, as it has become a pervasive feature of markets that may, 
among other things, amplify systemic risk. Examples include the following: 


- The International Organization of Securities Commissions (IOSCO) reported on the 
impact of new technologies including algorithmic trading on market surveillance, and 
made recommendations to consider, including for data collection and cross-border 
cooperation. °” 


- The Senior Supervisors’ Group (SSG), a forum for senior representatives of supervisory 
authorities from around the world, issued principles for supervisors to consider when 
assessing practices and key controls over algorithmic trading activities at banks. *® 


Some national regulators note that, from a supervisory perspective, firms developing 
algorithmic models based on AI and machine learning should have a robust development 
process in place. They need to ensure that possible risks are considered at every stage of the 
development process. This is particularly important in order to avoid market abuse and prevent 
the strategy from contributing to, or causing, disorderly market behaviour.*? This requirement 
is part of MiFID II, which will come into force in the first quarter of 2018 in Europe. There are 
similar requirements for algorithms imposed on certain regulated entities by a US securities 
self-regulatory organisation.” 


Similarly, the Basel Committee on Banking Supervision (BCBS) notes that a sound 
development process should be consistent with the firm’s internal policies and procedures and 
deliver a product that not only meets the goals of the users, but is also consistent with the risk 
appetite and behavioural expectations of the firm. In order to support new model choices, firms 
should be able to demonstrate developmental evidence of theoretical construction; behavioural 
characteristics and key assumptions; types and use of input data; numerical analysis routines 
and specified mathematical calculations; and code writing language and protocols (to replicate 
the model). Finally, it notes that firms should establish checks and balances at each stage of the 
development process.”! 





IOSCO (2011), “Regulatory Issues Raised by the Impact of Technological Changes on Market Integrity and Efficiency,” 
July; IOSCO (2013), “Technological Challenges to Effective Market Surveillance Issues and Regulatory Tools,” April 


Senior Supervisors’ Group (2015), “Algorithmic Trading Briefing Note,” April. 

These statements were made in discussions at the two workshops and in internal discussions in the FIN. 
90 See FINRA (2015), “FINRA Rule 3110 (Supervision),” June. 

9! See BCBS (2011), “Principles for the Sound Management of Operational Risk,” June. 
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5. Macro-financial analysis 


Widespread adoption of AI and machine learning could impact the financial system in a number 
of ways, depending on the nature of the application. From the perspective of economic growth, 
the application of AI and machine learning to financial services has potential to enhance the 
efficiency of the economy and to contribute to growth through the following mechanisms: ”? 


(a) Enhancing the efficiency of financial services: more efficient risk management of individual 
banks’ loan portfolio and insurers’ liabilities may benefit the aggregate system. AI and 
machine learning could help process information on the fundamental value of assets, thus 
allocating funds to investors and projects more effectively. Moreover, if AI and machine 
learning increase the speed and reduce the costs of payment and settlement transactions, for 
example by executing trades at times when there are available counterparties with 
corresponding demand, this may stimulate transactions for real economic activities. 


(b) Facilitating collaboration and realising new ‘economies of scope:’ Were AI and machine 
learning to facilitate collaboration between financial services and various industries, such 
as e-commerce and ‘sharing economy’ industries, this could realise new economies of scope 
and foster greater economic growth. For example, customer analysis based on transaction 
data attached to payment and settlement activities (for example, “who buys what, when, and 
where?”) would encourage cooperation between e-commerce and financial services. 


(c) Stimulating investments in AI and machine learning related areas: Many firms, including 
non-financial businesses, appear eager to apply AI and machine learning to their business. 
The growth in investments in AI and machine learning-related R&D can directly contribute 
to economy-wide investment and thus stimulate economic growth. 


From a macro-financial viewpoint, the short- to medium-term effects of the adoption of AI and 
machine learning on financial structure and markets could be more mixed. There are a number 
of potential effects on the systemic importance of market participants, the degree of 
concentration, and market vulnerabilities, which are elaborated below. 


5.1 Market concentration and systemic importance of institutions 


AI and machine learning may affect the type and degree of concentration in financial markets 
in certain circumstances. For instance, the emergence of a relatively small number of advanced 
third-party providers in AI and machine learning could increase concentration of some 
functions in the financial system. Similarly, access to big data could be a source of systemic 
importance, especially if firms are able to leverage their proprietary sources of big data to obtain 
substantial economies of scope. Finally, the most innovative technologies may be mainly 
affordable to large companies because the development of uses requires significant investments 
(for acquiring and maintaining the infrastructure and the skilled workers). 


For the possible impact of AI and machine learning on banks’ systemic importance, there are a 
number of key scenarios. If AI and machine learning ‘unbundle’ traditional banking services 


° Fora central bank’s view focusing on the impacts of AI on the economy more broadly, see Carolyn Wilkins (2017), “Blame 
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and entice new firms to offer financial services, this might reduce the systemic importance of 
individual large universal banks. These banks could focus on a more narrow set of activities, 
rather than continuing to offer universal services.” However, taken as a group, universal banks’ 
vulnerability to systemic shocks may grow if they increasingly depend on similar algorithms or 
data streams. On the other hand, if a large bank, which already has public trust, successfully 
adopts AI and machine learning so as to strengthen its market power, its systemic importance 
could increase. Whether other market participants provide similar services on competitive terms 
may also be affected by market entry costs and regulation. Thus, it is difficult to assess whether 
AI and machine learning would generally increase or decrease the degree of concentration. 


5.2 Potential Market Vulnerabilities 


Use of AI and machine learning for trading could impact the amount and degree of ‘directional’ 
trading. Under benign assumptions, the divergent development of trading applications by a 
wide range of market players could benefit financial stability. For example, if machine learning- 
powered robo-advisors give more customised advice to individuals, their investment activities 
may become more tailored to individual preferences and perhaps less correlated with other 
trading strategies. By reducing the barriers to entry for retail consumers to invest, these 
applications could also expand the investor base in capital markets. Similarly, the use of AI and 
machine learning for new and uncorrelated trading strategies by hedge funds could also result 
in greater diversity in market movements. More efficient processing of information could help 
to reduce price misalignments earlier and hence mitigate the build-up of macro-financial price 
imbalances. 


On the other hand, new trading algorithms based on machine learning may be less predictable 
than current rule-based applications and may interact in unexpected ways. To the extent that 
firms using AI or machine learning techniques can generate higher returns or lower trading 
costs, it is likely that incentives for adoption will increase. In the absence of data on the extent 
of market-wide use, market movements may be ascribed to AI and machine learning models, 
and interpretation of market shocks may be hampered. Finally, high frequency trading (HFT) 
applications of AI and machine learning could be new sources of vulnerabilities. If a similar 
investment strategy based on AI and machine learning is widely used in HFT, it might increase 
market volatility through large sales or purchases executed almost simultaneously.” 


Regarding leverage, liquidity, and maturity transformation, the adoption of AI and machine 
learning by financial market participants such as hedge funds and market makers may also have 
both positive and negative impacts. AI and machine learning could increase liquidity in 
financial markets through enhanced speed and efficiency of trading activities. AI and machine 
learning could be used to detect excessive risks and overly-complicated transactions and to 
design more effective hedging strategies for risk management by individual financial 


% See Hiroshi Nakaso (2016), “FinTech — Its Impacts on Finance, Economies and Central Banking,” speech, November 18. 





°4 See, e.g., FSB FinTech Issues Group (2017), p.46: “we have found no empirical evidence so far on convergence of robo- 
advisors’ algorithms or portfolios.” 
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institutions.” To the extent these tools enable the growth of new credit platforms to directly 
connect lenders and borrowers (broadly called FinTech credit),”° this could reduce reliance on 
bank loans, reduce banks’ leverage, and achieve a more diversified risk-sharing structure in the 
overall financial system. On the other hand, to the extent that market participants use AI and 
machine learning in order to minimise capital or margins or maximise expected returns on 
capital (within the constraints of regulations, and without paying due attention to risks), the use 
of AI and machine learning may increase risks. Specifically, it may allow for much tighter 
liquidity buffers, higher leverage, and faster maturity transformation than in cases where AI 
and machine learning had not been used for such optimisation. 


5.3 Networks and interconnectedness 


Applications of AI and machine learning may enhance the interconnectedness of financial 
markets and institutions in unexpected ways. Institutions’ ability to make use of big data from 
new sources may lead to greater dependencies on previously unrelated macroeconomic 
variables and financial market prices, including from various non-financial corporate sectors 
(e-commerce, sharing economy, etc.). As institutions find algorithms that generate uncorrelated 
profits or returns, there is a risk these will be exploited on a sufficiently wide scale that 
correlations actually increase. These potentially unforeseen interconnections will only become 
clear as technologies are actually adopted. 


More generally, greater interconnectedness in the financial system may help to share risks and 
act as a shock absorber up to a point. Yet the same factors could spread the impact of extreme 
shocks.” If a critical segment of financial institutions rely on the same data sources and 
algorithmic strategies, then under certain market conditions a shock to those data sources — or 
a new strategy exploiting a widely-adopted algorithmic strategy — could affect that segment as 
if it were a single node. This may occur even if, on the surface, the segment is made up of tens, 
hundreds, or even thousands of legally independent financial institutions. As a result, collective 
adoption of AI and machine learning tools may introduce new risks. 


5.4 Other implications of AI and machine learning applications 


AI and machine learning applications in insurance markets could reduce the degree of moral 
hazard and adverse selection — but could also undermine the risk pooling function of insurance. 
Moral hazard and adverse selection are inherent problems in insurance. Nonetheless, if AI and 
machine learning are used to continuously adjust insurance fees in accordance with changing 
behaviour of the policyholders, this may reduce moral hazard. If AI and machine learning are 
utilised to offer customised insurance policies reflecting detailed characteristics of each person, 
it may also decrease adverse selection. On the other hand, these uses may pose various new 
challenges. For example, the more accurate pricing of risk may lead to higher premiums for 
riskier consumers (such as in health insurance for individuals with a genetic predisposition to 


95 See the debates in the panel discussion entitled “FinTech and the Transformation of Financial Services” held in the 
International Monetary Fund on April 19, 2017. 


96 See CGFS and FSB (2017). 
°7 Andrew Haldane (2009), “Rethinking the financial network,” speech at the Financial Student Association, April. 
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certain diseases) and could even price some individuals out of the market. Even if innovative 
insurance pricing models are based on large data sets and numerous variables, algorithms can 
entail biases that can lead to non-desirable discrimination and even reinforce human prejudices. 
This warrants a societal discussion on the desired extent of risk sharing, how the algorithms are 
conceived, and which information is admissible.”® 


Meanwhile, AI and machine learning can continue to be a useful tool both for financial 
institutions (RegTech) and supervisors (SupTech). Many of the uses described in section 3.4 
could result in improvements in risk management, compliance, and systemic risk monitoring, 
while potentially reducing regulatory burdens. Yet, if a similar type of AI and machine learning 
is used without appropriately ‘training’ it or introducing feedback, reliance on such systems 
may introduce new risks. For example, if AI and machine learning models are used in stress 
testing without sufficiently long and diverse time series or sufficient feedback from actual stress 
events, there is a risk that users may not spot institution-specific and systemic risks in time. 
These risks may be pronounced especially if AI and machine learning are used without a full 
understanding of the underlying methods and limitations. 


Furthermore, as the current regulatory framework is not designed with the use of such tools in 
mind, some regulatory practices may need to be revised for the benefits of AI and machine 
learning techniques to be fully harnessed. For example, in MiFID II, where an obligation is 
placed on the firm to submit a report when a reportable event occurs, regulatory compliance is 
expected of the firm at all times. If AI and machine learning tools are used to deem if a particular 
activity is reportable or not, mistakes would still result in regulatory action, even if the tools 
can identify what information the regulators truly needs in order to reduce the risk of market 
disruption. In this regard, combining AI and machine learning with human judgment and other 
available analytical tools and methods may be more effective, particularly to facilitate causal 
analysis.” More generally, the greater adoption of AI, machine learning, and other 
technological advances in finance may benefit also from more of a ‘systems’ perspective in 
financial regulation to contribute to financial stability in an increasingly complex system.!° 


If optimisation solutions are adopted primarily by the private sector but not the public sector, 
there may be a risk that some individuals or firms may use them more successfully to ‘game’ 
regulatory rules or conduct regulatory arbitrage. 


6. Conclusions and implications for financial stability 


The use of AI and machine learning technology is changing the provision of some financial 
services. While data on the extent of adoption in various markets is quite limited, dialogue with 
market participants suggests that some segments of the financial system are actively employing 
AI and machine learning. These applications are thus currently more widely used than other 





% See IAIS (2017). 
9 Susan Athey (2017), “Beyond prediction: Using big data for policy problems,” Science 355(6324): 483-485. 


100 Andrei A. Kirilenko and Andrew W. Lo (2013), “Moore’s Law versus Murphy’s Law: Algorithmic Trading and Its 
Discontents,” Journal of Economic Perspectives, 27(2) 51-72; Susan Athey (2017), “Beyond prediction: Using big data 
for policy problems,” Science 355(6324): 483-485. 
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key FinTech innovations, such as distributed ledger technology or smart contracts. In particular, 
fraud detection, capital optimisation, and portfolio management applications appear to be 
growing rapidly. Most market participants expect that AI and machine learning will be adopted 
further. Because of this, it is important to start thinking about the financial stability implications 
now rather than after the potential implications have been realised. '°! The analysis is necessarily 
partial and will benefit from greater understanding of use cases over time. Moreover, many of 
the changes will not result in a material change to financial stability and hence fall outside the 
scope of this report. 


The use of AI and machine learning in financial services may bring key benefits for financial 
stability in the form of efficiencies in the provision of financial services and regulatory and 
systemic risk surveillance. The more efficient processing of information on credit risks and 
lower-cost customer interaction may contribute to a more efficient financial system. The 
internal (back-office) applications of AI and machine learning could improve risk management, 
fraud detection, and compliance with regulatory requirements, potentially at lower cost. In 
portfolio management, the more efficient processing of information from AI and machine 
learning applications could help to boost the efficiency and resilience of financial markets — 
reducing price misalignments earlier and (under benign assumptions) reducing crowded trades. 
Finally, with use cases by regulators and supervisors, there is potential to increase supervisory 
effectiveness and perform better systemic risk analysis in financial markets. 


At the same time, network effects and scalability of new technologies may in the future give 
rise to additional third-party dependencies. This could in turn lead to the emergence of new 
systemically important players. AI and machine learning services are increasingly being offered 
by a few large technology firms. Like in other platform-based markets, there may be value in 
financial institutions using similar third-party providers given these providers’ reputation, scale, 
and interoperability. There is the potential for natural monopolies or oligopolies. These 
competition issues — relevant enough from the perspective of economic efficiency — could be 
translated into financial stability risks if and when such technology firms have a large market 
share in specific financial market segments. These third-party dependencies and 
interconnections could have systemic effects if such a large firm were to face a major disruption 
or insolvency. 


Many current providers of AI and machine learning tools in financial services may fall outside 
the regulatory perimeter or may not be familiar with applicable law and regulation. Where 
financial institutions rely on third-party providers of AI and machine learning services for 
critical functions, and rules on outsourcing may not be in place or not be understood, these 
servicers and providers may not be subject to supervision and oversight. Similarly, if providers 
of such tools begin providing financial services to institutional or retail clients, this could entail 
financial activities taking place outside of the regulatory perimeter. 


The lack of interpretability or ‘auditability’ of AI and machine learning methods has the 
potential to contribute to macro-level risk if not appropriately supervised by microprudential 
supervisors. Many of the models that result from the use of AI or machine learning techniques 
are difficult or impossible to interpret. The lack of interpretability may be overlooked in various 
situations, including, for example, if the model’s performance exceeds that of more 





101 See CGFS and FSB (2017); FSB (2017). 
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interpretable models. Yet the lack of interpretability will make it even more difficult to 
determine potential effects beyond the firms’ balance sheet, for example during a systemic 
shock. Notably, many AI and machine learning developed models are being ‘trained’ in a period 
of low volatility. As such, the models may not suggest optimal actions in a significant economic 
downturn or in a financial crisis, or the models may not suggest appropriate management of 
long-term risks. 


Should there be widespread use of opaque models, it would likely result in unintended 
consequences. For example, if multiple firms develop trading strategies using AI and machine 
learning models but do not understand the models because of their complexity, it would be very 
difficult for both firms and supervisors to predict how actions directed by models will affect 
markets. When the models’ actions interact in the marketplace, it is quite possible that 
unintended, and possibly negative, consequences could result for financial markets. Similar 
unintended consequences may occur in applications aimed at credit scoring, capital 
optimisation, or cyber threat detection, where the build-up of risks may occur slowly. 


As with the use of any new product or service, there are important issues around the appropriate 
risk management and oversight of AI and machine learning. In discussions with FSB members 
for this report, industry representatives noted the challenges posed by conducting audits 
effectively, including sufficient skills in-house to understand and supervise AI and machine 
learning models. Beyond the staff operating these applications, key functions such as risk 
management and internal audit and the administrative management and supervisory body 
should be fit for controlling and managing the use of applications. Yet, the scarcity of resources 
with the required skills and knowledge can be an issue. 1%? On the supervisory side, auditing of 
models may require skills and expertise that supervisory institutions may not currently have. 
Some supervisors note a need to examine specifications developed in the scheduling and staging 
process of model development and to assess the governance structure around various stages of 
the model after its launch. !° 


Assessing AI and machine learning applications for risks, including adherence to any relevant 
protocols regarding data privacy, conduct risks, and cybersecurity, is important at this stage. It 
is important that progress in AI and machine learning applications is accompanied with further 
progress in the interpretation of algorithms’ outputs and decisions. Increased complexities of 
models may strain the abilities of developers and users to fully explain, and/or, in some 
instances, understand how they work. Efforts to improve the interpretability of AI and machine 
learning may be important conditions not only for risk management as noted above, but also 
for greater trust from the general public as well as regulators and supervisors in critical financial 
services. 


The uses of AI and machine learning should continue to be monitored. As the underlying 
technologies develop further, there is potential for more widespread use, beyond the use cases 
discussed in this report. It will be important to continue monitoring these innovations and to 
update this assessment in the future. 





102 FSB (2017), pp. 31-32. 


103 This statement derives from discussions at the two workshops and in internal discussions in the drafting team. 
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Glossary 
Algorithm: a set of computational rules to be followed to solve a mathematical problem. More 
recently, the term has been adopted to refer to a process to be followed, often by a computer. 


Artificial intelligence: the theory and development of computer systems able to perform tasks 
that traditionally have required human intelligence. 


Augmented intelligence: augmentation of human capabilities with technology, for instance by 
providing a human user with additional information or analysis for decision-making. 


Big data: a generic term that designates the massive volume of data that is generated by the 
increasing use of digital tools and information systems. 


Chatbots: virtual assistance programmes that interact with users in natural language. 


Cluster analysis: A statistical technique whereby data or objects are classified into groups 
(clusters) that are similar to one another but different from data or objects in other clusters. 


Deep learning: a subset of machine learning, this refers to a method that uses algorithms 
inspired by the structure and function of the brain, called artificial neural networks. 


FinTech: technologically enabled financial innovation that could result in new business 
models, applications, processes, or products with an associated material effect on financial 
markets and institutions and the provision of financial services. 


InsurTech: the application of FinTech for insurance markets. 


Internet of things: the inter-networking of physical devices, vehicles, buildings, and other 
items embedded with electronics, software, sensors, actuators, and network connectivity that 
enable these objects to collect and exchange data and send, receive, and execute commands. 


Machine learning: a method of designing a sequence of actions to solve a problem that 
optimise automatically through experience and with limited or no human intervention. 


Margin valuation adjustment: a method to determine the funding cost of the initial margin 
posted for a derivatives transaction. 





Natural Language Processing (NLP): An interdisciplinary field of computer science, artificial 
intelligence, and computation linguistics that focuses on programming computers and 
algorithms to parse, process, and understand human language. 


Open source: a designation for a computer programme in which underlying source code is 
freely available for redistribution and modification. 


RegTech: any range of applications of FinTech for regulatory and compliance requirements 
and reporting by regulated financial institutions. This can also refer to firms that offer such 
applications, and in some cases can encompass SupTech (see below). 


Reinforcement learning: a subset of machine learning in which an algorithm is fed an 
unlabelled set of data, chooses an action for each data point, and receives feedback (perhaps 
from a human) that helps the algorithm learn. 


Robo-advisors: applications that combine digital interfaces and algorithms, and can also 

include machine learning, in order to provide services ranging from automated financial 

recommendations to contract brokering to portfolio management to their clients, without or 
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with very limited human intervention. Such advisors may be standalone firms and platforms, 
or can be in-house applications of incumbent financial institutions. 


Social trading: a range of trading platforms that allow users to compare trading strategies or 
copy the trading strategy of other investors. The latter is often referred to as ‘copy trading’ or 
‘mirror investing.’ 


SupTech: applications of FinTech by supervisory authorities. 


Supervised learning: a subset of machine learning in which an algorithm is fed a set of 
‘training’ data that contains labels on the observations. 


Terabyte: a unit of data storage, equal to one trillion (10!) bytes, or 1,000 gigabytes. 


Tonality analysis: a method to gauge the negativity of a piece of text by counting terms with 
a negative connotation. 


Topic modelling: a method of unsupervised learning lets the data define key themes in text. 


Unsupervised learning: a subset of machine learning in which the data provided to the 
algorithm does not contain labels. 


Zettabyte: a unit of data storage, equal to one sextillion (107') bytes, one trillion gigabytes, or 
one billion terabytes. 
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Annex A: Legal issues around AI and machine learning 








Al and machine learning present a range of legal issues relating to privacy and data protection, 
consumer protection, anti-discrimination and liability issues, and cross-border issues. 


The issues around data privacy relate to the ability to access the data being processed by AI and 
machine learning tools. While big data are widely used to generate profits, they can only do so 
with technology that converts the data into relevant services.' In financial services, AI and 
machine learning applications usually depend on access to, and use of, large amounts of data in 
a ‘life cycle’ that includes data collection, data compilation and consolidation, data mining and 
analytics. The applicability of laws and regulations will be generally fact-driven and 
jurisdiction-specific. But certain legal issues are being commonly evaluated in the context of 
the use of AI and machine learning with big data, including the applicability of data ownership 
rights and data privacy protections and cross-border flows of data. Regulatory authorities are 
accelerating efforts to understand the implications for the financial sector.’ 


Data ownership rights and protections are being revised in several jurisdictions. Among OECD 
members, many have privacy protection laws, and the OECD has guidelines on the protection 
of privacy and cross-border uses.* The European Union (EU) recently enacted a General Data 
Protection Regulation (GDPR), due to come into force in 2018. Especially relevant with respect 
to the use of AI and machine learning are Article 11, which provides a right to “an explanation 
of the decision reached after [algorithmic] assessment,” and allied articles providing for similar 
disclosures.” Other key articles relating to AI and machine learning are Article 9, which 
prohibits the processing of “special [sensitive] categories of personal data” as defined; Article 
22, which provides for a data subject’s qualified right not to be subject to a decision with legal 
or significant consequences based solely on automated processing; and Article 24, which 
provides that decisions shall not be based on special categories of personal data.° 


Ownership of intellectual property in personal data and data protections may be of particular 
relevance when a data subject wishes to move their profile to a different data processor, or when 
a regulator steps in to transfer a service to another provider. In both cases, having intellectual 
property arrangements that conform with regulatory and client expectations may be challenging 
given the complexity of the subject area, including cross-border issues (see below). 


Meanwhile, some authorities are exploring if consumers should have the ability to understand 
complex modelling techniques for credit systems.’ New tools developed to improve 
interpretability of AI and machine learning models can aid firms and policymakers.’ Consumer 
protection, anti-discrimination and liability issues are areas of emerging focus and future work, 
including on the use of big data analytics in risk management and macroprudential surveillance. 
A variety of consumer protection laws already apply to big data practices that might include the 
use of AI and machine learning, such as fair credit reporting, equal opportunity laws, and fair 
trade practices. Yet not all of these regulations may address AI and machine learning techniques 
using large quantities of data in digital form. 


Anti-discrimination laws in particular may be relevant to AI and machine learning techniques. 
Machine learning models may result in discriminatory practices that may be unlawful, even 
where characteristics such as racial or gender information is not input directly; the development 
of non-discriminatory data mining techniques is an active but unsolved area of research.” 
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Regarding legal liability, there may be questions on the allocation of responsibility among 
suppliers, operators and users of AI and machine learning systems — for example the 
responsibility of a manufacturer or distributor of a financial product that is based on third party 
data input devices or algorithms.!° There are difficult liability issues, including the extent to 
which humans may be entitled to rely on expert systems in a wide range of settings. Such 
liability issues will become increasingly important as artificial agents perform a broader range 
of tasks currently performed by humans, with the potential for mistakes and for legal disputes 
around damages.!! 


Finally, the growth of AI and machine learning applications could lead to cross-border issues. 
Currently, the development of these technologies in finance is concentrated in a small number 
of countries, while adoption may occur at financial institutions around the world. Regulators 
should keep in mind that cross-border supervision, cooperation and investigation and other 
regulatory issues may be expected to arise with AI and machine learning applications active 
across jurisdictions. 


Nobuchika Mori (2017), “Will FinTech create shared values?” speech at Columbia Business School conference, May. 





2 Defined in EIOPA (2017), “Opinion of the Occupational Pensions Stakeholder Group on JC Big Data,” EIOPA-OPSG-17- 
06 15, March, pp. 6-7. See also U.S. Federal Trade Commission (2016), “Big Data: A Tool for Inclusion or Exclusion,” 
January, p. 3. 


3 See EIOPA (2017); U.S. Federal Register (2017), Vol. 82, No.33, and Bureau of Consumer Financial Protection: Docket 
No. CFPB Notice and Request for Information Regarding Use of Alternative Data and Modelling Techniques in the Credit 
Process, February 21, 2017 (“CFPB RFI’); European Banking Authority (2017), “Report on innovative uses of consumer 
data by financial institutions, June. See also FSB FinTech Issues Group (2017), p. 19. 


4 OECD (2013), “Guidelines on the Protection of Privacy and Transborder Flows of Personal Data,” July. 


For instance, Articles 13, 14, and 15 require disclosure of the existence of automated decision-making, including profiling, 
referred to in Article 22(1) and (4) and, at least in those cases, “meaningful information about the logic involved,” as well 
as the significance and the envisaged consequences of such processing for the data subject. 


€ Note that Articles 9, 22 and 24 are all subject to exceptions. See Sandra Wachter, Brent Mittelstadt, and Luciano Floridi 
(2017), “Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection 
Regulation,” International Data Privacy Law, Forthcoming; and Bryce Goodman and Seth Flaxman (2016), “European 
Union regulations on algorithmic decision-making and a ‘right to explanation,”” paper presented at 2016 ICML Workshop 
on Human Interpretability in Machine Learning (WHI 2016), New York. Wachter et al. argue that these provisions confer 
no right to an ex-post explanation of decisions, though ex-post explanations may be crafted through jurisprudence or EDPB 
work. Goodman and Flaxman on the other hand argue the law will also effectively create a “right to explanation,” whereby 
a user can ask for an explanation of an algorithmic decision that was made about them. 








7 Michael Gordon and Vaughn Stewart (2017), “Insights on Alternative Data use on Credit Scoring,” CPFB Law360, May. 


See Pang Wei Koh and Percy Liang (2017), “Understanding Black-box Predictions via Influence Functions,” Proceedings 
of the 34" International Conference on Machine Learning, Sydney; Marco Tulio Ribiero, Sameer Singh and Carlos Guestrin 
(2016), “ Why Should I Trust You?’ Explaining the Predictions of Any Classifier,” arXiv:1602.04938v3; and Fast Forward 
Labs (2017), “New Research on Interpretability,” August. 








See Bettina Berendt and Sören Preibusch (2014), “Better decision support through exploratory discrimination-aware data 
mining: foundations and empirical evidence,” Artificial Intelligence and Law 22 (2): 175-209; Indré Zliobaité (2017), 
“Measuring discrimination in algorithmic decision making,” Data Mining and Knowledge Discovery 31(4): 1060-1089; 
and Bruno Lepri, Jacopo Staiano, David Sangokoya, Emmanuel Letouze and Nuria Oliver (2016), “The Tyranny of Data? 
The Bright and Dark Sides of Data-Driven Decision-Making for Social Good,” working paper, December. 


10 See EIOPA (2017), pp. 6-7. 








Laurence White and Samir Chopra (2011), A Legal Theory for Autonomous Artificial Agents, University of Michigan Press, 
chapter 4. 
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Annex B: AI ethics 








The rapid development of AI has inspired both hope and concern about rapidly growing 
sophistication and capabilities. These discussions extend far beyond AI in financial services 
and have inspired new research in philosophy on ‘machine ethics,’ which concerns itself with 
ethical norms in the behaviour of artificial agents.’ As artificial agents take on responsibilities 
in areas, such as executing financial transactions, driving cars, and controlling complex systems 
of devices, there are concerns about how to ensure ethical behaviour in the interests of users. 
Recently, Bostrom (2014) has discussed the implications of ‘superintelligence,’ or AI systems 
whose capabilities surpass those of humans, potentially resulting in new challenges and 
unintended consequences.” In 2015, a diverse group of technology and science luminaries 
including Stephen Hawking, Bill Gates, and Elon Musk signed an open letter urging more 
research into the benefits and risks of AI and another on the dangers of autonomous weapons.’ 


One issue is that AI and machine learning may reinforce biases. Some commentators point to 
the potential of big data analytics to entrench existing biases in college applications, job 
selection, prison sentencing, and credit provision. For example, unsupervised learning 
algorithms may show fewer high-level job vacancies to female applicants, and sentencing 
algorithms may be harsher for ethnic minorities. Yet depending on programming choices, they 
may also present opportunities to reduce such discrimination.’ Other writers note that the use 
of machine learning to filter news and social contacts or programming choices which 
disadvantage some languages may have far-reaching long-run consequences.° 


There are a number of initiatives to further research on AI ethics and the ethical use of AI. For 
example, in September 2016, a group of large tech firms (Amazon, Facebook, Google, IBM 
and Microsoft) founded the Partnership on Artificial Intelligence, which aims to “develop and 
share best practices on AI,” “advance public understanding” and “identify and foster... AI for 
socially beneficial purposes.” Apple, eBay, SalesForce, Sony, the ACLU, Human Rights 
Watch, UNICEF, and a number of other organisations have since joined the partnership, and it 
has launched research around thematic pillars including “AI, labour and the economy” and 
“safety-critical AI”.” Concurrently, the European parliament has debated the legal status of 
autonomous agents.® Finally, machine ethics remains a field of active research, which may yet 
yield further insights for broader public policy. 


' Michael Anderson and Susan Leigh Anderson (eds., 2011), Machine Ethics, Cambridge University Press; 
Wendell Wallach and Colin Allen (2008), Moral Machines: Teaching Robots Right from Wrong. Oxford 
University Press. 

? Nick Bostrom (2014), Superintelligence: Paths, Dangers, Strategies, Oxford: Oxford University Press. 

3 Future of Life Institute (2015), “Research Priorities for Robust and Beneficial Artificial Intelligence: an Open 
Letter,” January; --- (2015), “Autonomous Weapons: an Open Letter from AI & Robotics Researchers,” July. 

4 Cathy O’Neil (2016), Weapons of Math Destruction: How Big Data Increases Inequality and Threatens 
Democracy, London: Allen Lane. 

5 Executive Office of the President (2016), “Big Data: A Report on Algorithmic Systems, Opportunity, and Civil 
Rights,” Washington, D.C., May. 

ê Robert Munro (2015), “The threat from AI is real, but everyone has it wrong,” Operational Database Management 
Systems, August. 

7 Partnership on AI (2017), “Partnership on AI Strengthens Its Network of Partners and Announces First 
Initiatives,” May. 

8 European Parliament Committee on Legal Affairs (2017), “Draft Report with recommendations to the 
Commission on Civil Law Rules on Robotics,” January. 
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ARTICLE 


ACCOUNTABLE ALGORITHMS 


JOSHUA A. KROLL, JOANNA HUEY, SOLON BAROCAS, EDWARD W. 
FELTEN, JOEL R. REIDENBERG, DAVID G. ROBINSON 
& HARLAN YUt 


Many important decisions historically made by people are now made by 
computers. Algorithms count votes, approve loan and credit card applications, target 
citizens or neighborhoods for police scrutiny, select taxpayers for IRS audit, grant or 
deny immigration visas, and more. 

The accountability mechanisms and legal standards that govern such decision 
processes have not kept pace with technology. The tools currently available to 
policymakers, legislators, and courts were developed to oversee human decisionmakers 
and often fail when applied to computers instead. For example, how do you judge the 
intent of a piece of software? Because automated decision systems can return potentially 
incorrect, unjustified, or unfair results, additional approaches are needed to make 
such systems accountable and governable. This Article reveals a new technological 
toolkit to verify that automated decisions comply with key standards of legal fairness. 

We challenge the dominant position in the legal literature that transparency will 
solve these problems. Disclosure of source code is often neither necessary (because of 
alternative techniques from computer science) nor sufficient (because of the issues 
analyzing code) to demonstrate the fairness of a process. Furthermore, transparency 
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may be undesirable, such as when it discloses private information or permits tax 
cheats or terrorists to game the systems determining audits or security screening. 

The central issue is how to assure the interests of citizens, and society as a whole, 
in making these processes more accountable. This Article argues that technology is 
creating new opportunities—subtler and more flexible than total transparency—to 
design decisionmaking algorithms so that they better align with legal and policy 
objectives. Doing so will improve not only the current governance of automated 
decisions, but also—in certain cases—the governance of decisionmaking in general. 
The implicit (or explicit) biases of human decisionmakers can be difficult to find and 
root out, but we can peer into the “brain” of an algorithm: computational processes 
and purpose specifications can be declared prior to use and verified afterward. 

The technological tools introduced in this Article apply widely. They can be used 
in designing decisionmaking processes from both the private and public sectors, and 
they can be tailored to verify different characteristics as desired by decisionmakers, 
regulators, or the public. By forcing a more careful consideration of the effects of 
decision rules, they also engender policy discussions and closer looks at legal standards. 
As such, these tools have far-reaching implications throughout law and society. 

Part I of this Article provides an accessible and concise introduction to 
foundational computer science techniques that can be used to verify and demonstrate 
compliance with key standards of legal fairness for automated decisions without 
revealing key attributes of the decisions or the processes by which the decisions were 
reached. Part II then describes how these techniques can assure that decisions are 
made with the key governance attribute of procedural regularity, meaning that 
decisions are made under an announced set of rules consistently applied in each case. 
We demonstrate how this approach could be used to redesign and resolve issues with 
the State Department’s diversity visa lottery. In Part III, we go further and explore 
how other computational techniques can assure that automated decisions preserve 
fidelity to substantive legal and policy choices. We show how these tools may be used 
to assure that certain kinds of unjust discrimination are avoided and that automated 
decision processes behave in ways that comport with the social or legal standards that 
govern the decision. We also show how automated decisionmaking may even 
complicate existing doctrines of disparate treatment and disparate impact, and we 
discuss some recent computer science work on detecting and removing discrimination 
in algorithms, especially in the context of big data and machine learning. And lastly, 
in Part IV, we propose an agenda to further synergistic collaboration between 
computer science, law, and policy to advance the design of automated decision 
processes for accountability. 
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INTRODUCTION 


Many important decisions that were historically made by people are now 
made by computer systems!: votes are counted; voter rolls are purged; loan 
and credit card applications are approved;2 welfare and financial aid decisions 
are made;3 taxpayers are chosen for audits; citizens or neighborhoods are 
targeted for police scrutiny;4 air travelers are selected for search;5 and visas 
are granted or denied. The efficiency and accuracy of automated decisionmaking 
ensures that its domain will continue to expand. Even mundane activities now 
involve complex computerized decisions: everything from cars to home 
appliances now regularly executes computer code as part of its normal operation. 

However, the accountability mechanisms and legal standards that govern 
decision processes have not kept pace with technology. The tools currently 
available to policymakers, legislators, and courts were developed primarily to 
oversee human decisionmakers. Many observers have argued that our current 
frameworks are not well-adapted for situations in which a potentially incorrect,6 
unjustified, or unfair’ outcome emerges from a computer. Citizens, and society 
as a whole, have an interest in making these processes more accountable. If 
these new inventions are to be made governable, this gap must be bridged. 


1 In this Article, we use the term “computer system” where others have used the term 
“algorithm.” See, e.g., FRANK PASQUALE, THE BLACK BOX SOCIETY: THE SECRET ALGORITHMS 
THAT CONTROL MONEY AND INFORMATION (2015). This allows us to separate the concept of a 
computerized decision from the actual machine that effects it. See infra note 14 for a more detailed 
explanation, 

2 See, e.g., Calyx - More Than Just an LOS, CALYX SOFTWARE (Mar. 2013), http://www.calyx 
software.com/company/newsletters/13-03.html [https://perma.cc/E93L-8UGD] (noting that Calyx 
offers clients an automated underwriting system to vet loan applications for approval against 
predetermined guidelines). 

3 See Virginia Eubanks, Caseworkers v. Computers, POPTECH (Dec. 11, 2013, 3:10 PM), http:// 
virginiaeubanks.wordpress.com/2013/12/11/caseworkers-vs-computers [https://perma.cc/37VG-GQC6] 
(describing and critiquing several states’ efforts to automate welfare eligibility determinations). 

4 See DAVID ROBINSON, HARLAN YU & AARON RIEKE, CIVIL RIGHTS, BIG DATA, AND OUR 
ALGORITHMIC FUTURE 18-19 (2014), http://bigdata.fairness.io/wp-content/uploads/2014/11/Civil_ 
Rights Big Data_and_Our_Algorithmic-Future_v1.1.pdf [https://perma.cc/UL3G-3MQ]] (describing 
the Chicago Police Department’s “‘Custom Notification Program, which sends police (or sometimes 
mails letters) to peoples’ homes to offer social services and a tailored warning”). 

5 See Notice of Modified Privacy Act System of Records, 78 Fed. Reg. 55,270, 55,271 (Sept. 10, 
2013) (“[T]he passenger prescreening computer system will conduct risk-based analysis of passenger 
data.... TSA will then review this information using intelligence-driven, risk-based analysis to 
determine whether individual passengers will receive expedited, standard, or enhanced screening. . ..”). 

6 See Danielle Keats Citron, Technological Due Process, 85 WASH. U. L. REV. 1249, 1256 (2008) 
(describing systemic errors in the automated eligibility determinations for federal benefits programs). 

7 See id. at 1256-57 (noting the “crudeness” of algorithms designed to identify potential 
terrorists that yield a high rate of false positives). 

8 See Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact, 104 CALIF. L. REV. 671, 
677 (2016) (“[D]ata mining holds the potential to unduly discount members of legally protected 


classes and to place them at systematic relative disadvantage.”). 
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In this Article, we describe how authorities can demonstrate—and how 
the public at large and oversight bodies can verify—that automated decisions 
comply with key standards of legal fairness. We consider two approaches: ex 
ante approaches aiming to establish that the decision process works as expected 
(which are commonly studied by technologists and computer scientists), and ex 
post approaches once decisions have been made, such as review and oversight 
(which are common in existing governance structures). Our proposals aim to 
use the tools of the first approach to guarantee that the second approach can 
function effectively. Specifically, we describe how technical tools for verifying 
the correctness of computer systems can be used to ensure that appropriate 
evidence exists for later oversight. 

We begin with an accessible and concise introduction to the computer 
science concepts on which our argument relies, drawn from the fields of 
software verification, testing, and cryptography. Our argument builds on the 
fact that technologists can and do verify for themselves that software systems 
work in accordance with known designs. No computer system is built and 
deployed in the world shrouded in total mystery.9 While we do not advocate 
any specific liability regime for the creators of computer systems, we outline 
the range of tools that computer scientists and other technologists already 
use, and show how those tools can ensure that a system meets specific policy 
goals. In particular, while some of these tools provide assurances only to the 
system’s designer or operator, other established methods could be leveraged to 
convince a broader audience, including regulators or even the general public. 

The tools available during the design and construction of a computer 
system are far more powerful and expressive than those that can be bolted on 
to an existing system after one has been built. We argue that, in many 
instances, designing a system for accountability can enable stakeholders to 
reach accountability goals that could not be achieved by imposing new 
transparency requirements on existing system designs. 

We show that computer systems can be designed to prove to oversight 
authorities and the public that decisions were made under an announced set of 
rules consistently applied in each case, a condition we call procedural regularity. 
The techniques we describe to ensure procedural regularity can be extended to 
demonstrate adherence to certain kinds of substantive policy choices, such as 
blindness to a particular attribute (e.g., race in credit underwriting). Procedural 
regularity ensures that a decision was made using consistently applied standards 


9 Although some machine learning systems produce results that are difficult to predict in advance 
and well beyond traditional interpretation, the choice to field such a system instead of one which can 
be interpreted and governed is itself a decision about the system’s design. While we do not advocate 
that any approach should be forbidden for any specific problem, we aim to show that advanced tools 
exist that provide the desired functionality while also permitting oversight and review. 
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and practices. It does not, however, guarantee that such practices are themselves 
good policy. Ensuring that a decision procedure is well justified or relies on 
sound reasoning is a separate challenge from achieving procedural regularity. 
While procedural regularity is a well-understood and generally desirable 
property for automated and nonautomated governance systems alike, it is 
merely one principle around which we can investigate a system’s fairness. 

It is common, for example, to ask whether a computer system avoids 
certain kinds of unjust discrimination, even when such systems are blind to 
certain attributes (e.g., gender in automated hiring decisions). We later 
expand our discussion and show how emerging computational techniques can 
assure that automated decisions satisfy other notions of fairness that are not 
merely procedural, but actively consider a system’s effects. We describe in 
particular detail techniques for avoiding discrimination, even in machine 
learning systems that derive their decision rules from data rather than from 
code written by a programmer. And finally, we propose next steps to further 
the emerging and critically important collaboration between computer 
scientists and policymakers. 

Legal scholars have argued for twenty years that automated processing 
requires more transparency, but it is far from obvious what form such 
transparency should take. Perhaps the most obvious approach is to disclose a 
system’s source code, but this is at best a partial solution to the problem of 
accountability for automated decisions. The source code of computer systems 
is illegible to nonexperts. In fact, even experts often struggle to understand 
what software code will do, as inspecting source code is a very limited way of 
predicting how a computer program will behave. Machine learning, one 
increasingly popular approach to automated decisionmaking, is particularly 
ill-suited to source code analysis because it involves situations where the 
decisional rule itself emerges automatically from the specific data under 
analysis, sometimes in ways that no human can explain.!2 In this case, source 
code alone teaches a reviewer very little, since the code only exposes the 
machine learning method used and not the data-driven decision rule. 


10 See, e.g., Citron, supra note 6, at 1253 (describing automated decisionmaking as “adjudicat[ion] 
in secret”); Paul Schwartz, Data Processing and Government Administration: The Failure of the American 
Legal Response to the Computer, 43 HASTINGS L.J. 1321, 1323-25 (1992) (“So long as government 
bureaucracy relies on the technical treatment of personal information, the law must pay attention to 
the structure of data processing . . . . There are three essential elements to this response: structuring 
transparent data processing systems; granting limited procedural and substantive rights ... and 
creating independent governmental monitoring of data processing systems.” (emphasis omitted)). 

11 See infra subsection I.A.1 (discussing static analysis). 

12 See Stanford Univ., Machine Learning, COURSERA, https://www.coursera.org/learn/ 
machine-learning/home/info [https://perma.cc/L7KF-CDYa4] (“Machine learning is the science of 
getting computers to act without being explicitly programmed.”). 


2017] Accountable Algorithms 639 


Moreover, in many of the instances that people care about, full 
transparency will not be possible. The process for deciding which tax returns 
to audit, or whom to pull aside for secondary security screening at the airport, 
may need to be partly opaque to prevent tax cheats or terrorists from gaming 
the system. When the decision being regulated is a commercial one, such as 
an offer of credit, transparency may be undesirable because it defeats the 
legitimate protection of consumer data, commercial proprietary information, 
or trade secrets. Finally, when an explanation of how a rule operates requires 
disclosing the data under analysis and those data are private or sensitive (e.g., 
in adjudicating a commercial offer of credit, a lender reviews detailed 
financial information about the applicant), disclosure of the data may be 
undesirable or even legally barred. 

Furthermore, making the rule transparent—whether through source code 
disclosure or otherwise—may still fail to resolve the concerns of many 
participants. No matter how much transparency surrounds a rule, people can 
still wonder whether the disclosed rule was actually used to reach a decision 
in their own cases. Particularly where an element of randomness is involved 
in the process, a person audited or patted down may wonder: was I really 
chosen by the rule, or has some bureaucrat singled me out on a whim? But 
full disclosure of how particular decisions were reached is often unattractive 
because the decisions themselves often incorporate sensitive health, financial, 
or other private information either as input, output, or both (for example, an 
individual’s tax audit status may be sensitive or protected on its own, but it 
may also imply details about that individual’s financial data). 

Even full disclosure of a decision’s provenance to that decision’s subject can 
be problematic. Most individuals are ill-equipped to review how computerized 
decisions are made, even if those decisions are reached transparently. Further, 
the purpose of computer-mediated decisionmaking is to bring decisions an 
element of scale, where the same rules are ostensibly applied to a large 
number of individual cases or are applied extremely quickly. Individuals 
auditing their own decisions (or experts assisting them) would be both 
inundated with the need to review the rules applied to them and often able 
to generalize their conclusions to the results of others, raising the same 
disclosure concerns described above. That is, while transparency of a rule 
makes reviewing the basis of decisions more possible, it is not a substitute for 
individualized review of particular decisions.%3 


13 Even when experts can pool investigative effort across many decisions, there is no guarantee 
that the basis for decisions will be interpretable or that problems of fairness or even overt special 
treatment for certain people will be discovered. Further, a regime based on individuals auditing their 
own decisions cannot adequately address departures from an established rule, which favor the 
individual auditing her own outcome, or properties of the rule, which can only be examined across 
individuals (such as nondiscrimination). 
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Fortunately, technology is creating new opportunities—more subtle and 
flexible than total transparency—to make automated decisionmaking more 
accountable to legal and policy objectives. Although the current governance 
of automated decisionmaking is underdeveloped, computerized processes can 
be designed for governance and accountability. Doing so will improve not only 
the current governance of computer systems, but also—in certain cases—the 
governance of decisionmaking in general. 

This Article argues that in order for a computer system to function in an 
accountable way—either while operating an important civic process or merely 
engaging in routine commerce—accountability must be part of the system’s 
design from the start. Designers of such systems, and the nontechnical 
stakeholders who often oversee or control system design, must begin with 
oversight and accountability in mind. We offer examples of currently available 
tools that could aid in that design, as well as suggestions for dealing with the 
apparent mismatch between policy ambiguity and technical precision. 

In Part I of this Article, we provide an accessible introduction to how 
computer scientists build and evaluate computer systems and the software 
and algorithms! that comprise them. In particular, we describe how computer 


14 In this Article, we limit our use of the word “algorithm” to its usage in computer science, where 
it refers to a well-defined set of steps for accomplishing a certain goal. In other contexts, where other 
authors have used the term “algorithm,” we describe “automated decision processes” reflecting “decision 
policies” implemented by pieces of “software,” all comprising “computer systems.” Our adoption of the 
phrase “computer systems” was suggested by (and originally due to) Helen Nissenbaum, and we are 
grateful for the precision it provides. See generally Batya Friedman & Helen Nissenbaum, Bias in 
Computer Systems, 14 ACM TRANSACTIONS ON INFO. SYSTEMS 330 (1996). 

The term “algorithm” is assigned disparate technical meanings in the literatures of computer 
science and other fields. The computer scientist Donald Knuth famously defined algorithms as 
separate from mathematical formulae in that (1) they must “always terminate after a finite number 
of steps;” (2) “[eJach step of an algorithm must be precisely defined; the actions to be carried out 
must be rigorously and unambiguously specified for each case;” (3) input to the algorithm is 
“quantities that are given to it initially before the algorithm begins;” (4) an algorithm’s output is 
“quantities that have a specified relation to the inputs;” and (5) the operations to be performed in 
the algorithm “must all be sufficiently basic that they can in principle be done exactly and in a finite 
length of time by someone using pencil and paper.” 1 DONALD E. KNUTH, THE ART OF COMPUTER 
PROGRAMMING: FUNDAMENTAL ALGORITHMS 4-6 (1968). Similarly and more simply, a widely used 
computer science textbook defines an algorithm as “any well-defined computational procedure that 
takes some value, or set of values, as input and produces some value, or set of values, as output.” THOMAS 
H. CORMEN ET AL., INTRODUCTION TO ALGORITHMS 10 (2d ed. 2001). 

By contrast, communications scholar Christian Sandvig says that 


ac, 


algorithm’ refers to the 
overall process” by which some human actor uses a computer to do something, including decisions 
made by humans as to what the computer should do, choices made during implementation, and even 
choices about how algorithms are represented and marketed to the public. Christian Sandvig, Seeing 
the Sort: The Aesthetic and Industrial Defense of “The Algorithm,” MEDIA-N, http://median.newmedia 
caucus.org/art-infrastructures-information/seeing-the-sort-the-aesthetic-and-industrial-defense-of- 
the-algorithm [https://perma.cc/29E4-S44S]. Sandvig argues that even algorithms as simple as 
sorting “have their own public relations” and are inherently human in their decisions. Jd. 
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scientists evaluate a program to verify that it has desired properties and 
discuss the value of randomness in the construction of many computer 
systems. We characterize what sorts of properties of a computer system can be 
tested and describe one of the fundamental truths of computer science—that 
there are some properties of computer systems which cannot be tested 
completely. We observe that computer systems fielded in the real world are 
(or at least should be) tested regularly during creation, deployment, and 
operation, merely to establish that they are actually functional. 

Part II examines how to design computer systems for procedural regularity, 
a key governance principle enshrined in law and public policy in many 
societies. We consider how participants, decision subjects, and observers can be 
assured that each individual decision was made according to the same 
procedure—for example, how observers can be assured that the decisionmaker 
is not choosing outcomes on a whim while merely claiming to follow an 
announced rule. We describe why mere disclosure of a piece of source code 
can be impractical or insufficient for these ends. Indeed, without full 
transparency—including source code, input data, and the full operating 
environment of the software—even the disclosure of audit logs showing what 
a program did while it was running provides no guarantee that the disclosed 
information actually reflects a computer system's behavior.!5 In order to move 
beyond the need for full transparency, we focus on tools that can communicate 
partial information about secret processes, so that accountability and 
oversight continue to function even when policy interests, personal privacy, 
trade secrets, or other concerns protect a computer system, a piece of 
software, its inputs, its outputs, or its environment from disclosure. Putting 
it all together, we provide an illustrative example of how to redesign an 
existing, legally mandated automated decisionmaking system—the State 
Department’s Diversity Visa Lottery—so that it is provably accountable. 


Another communications scholar, Nicholas Diakopoulos, defines algorithms in the narrow sense 
as “a series of steps undertaken in order to solve a particular problem or accomplish a defined 
outcome,” but also considers them in the broad sense, saying that “algorithms can arguably make 
mistakes and operate with biases,” which does not make sense for the narrower technical definition. 
Nicholas Diakopoulos, Algorithmic Accountability: Journalistic Investigation of Computational Power 
Structures, 3 DIGITAL JOURNALISM 398, 398, 400 (2015). This confusion is common in much of the 
literature on algorithms and accountability, which we describe throughout this Article. To avoid 
confusion, this Article adopts the precise definition of the word “algorithm” from computer science 
and, following Friedman and Nissenbaum, refers to the broader concept of an automated system 
deployed in a social or human context as a “computer system.” 

15 The environment of a computer system includes anything it might interact with. For 
example, an outside observer will need to know what other software was running on a particular 
computer to ensure that nothing modified the behavior of the disclosed program. Some programs 
also observe (and change their behavior based on) the state of the computer they are running on 
(such as which files were or were not present or what other programs were running), the time they 
were run, or even the configuration of hardware on the system on which they were run. 
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Part III considers the broader question of how to assess a computer 
systems compliance with policy principles that go beyond procedural 
regularity. These broader properties include determining whether automated 
decision systems treat people (including protected groups) in ways that 
comport with the social or legal standards that govern the decision being 
made.1¢ This broadening raises the issue of translating a policy principle into 
a property of the system. Certain substantive policy choices translate easily: 
for example, prespecified rules such as blindness to a sensitive attribute.” 
Defining other policy objectives, such as a general notion of nondiscrimination, 
however, is a more complicated and fraught affair, particularly when systems 
rely on machine learning rather than decision rules explicitly predetermined by 
humans. We explore in particular the discriminatory effect that automated 
decisionmaking can have, noting real-world examples of newfound risks and 
describing some system properties that may align with policy goals. Finally, we 
observe how automated decisionmaking may complicate the existing doctrine 
of disparate treatment and disparate impact. 

Part IV concludes by calling for increased collaboration between computer 
scientists and policymakers to develop and apply technical tools for the 
governance of computer systems. Given the ever-widening reach of automated 
decisions, computer scientists need to understand the policy challenges of 
oversight, and policymakers need to understand where new and emerging 
software tools can help address those challenges. We offer recommendations 
for bridging the gap between technologists’ desire for specificity and the 
policy process’s need for ambiguity. As a first step, we urge policymakers to 
recognize that accountability is feasible even when the details of a computer 
system are not fully known or must be kept secret. We also argue that the 
ambiguities, contradictions, and uncertainties of the policy process need not 
discourage computer scientists from engaging constructively in it. 


I. HOW COMPUTER SCIENTISTS BUILD AND EVALUATE SOFTWARE 


Fundamentally, computers are general purpose machines that can be 
programmed to do any computational task, though they lack the desirable 
specificity and limitations of physical devices.!8 Engineers often seek strong 


16 This type of evaluation depends upon having already verified procedural regularity: if it 
cannot be determined that a particular algorithm was used to make a decision, it is fruitless to try 
to verify properties of that algorithm. 

17 A concrete example would be the requirement that a decision only account for certain 
information for certain purposes, as in a system for screening job applicants that is allowed to take 
the gender of applicants as input, but only for the purpose of keeping informational statistics and 
not for making screening decisions. 

18 For example, hydraulically operated control surfaces in a vehicle will telegraph resistance to 
the operator when they are close to a dangerous configuration, but the same controls operated by a 
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digital evidence that a computer system is working as intended. Such evidence 
may be persuasive for the systems creator or operator, for a predesignated 
group of receivers such as an oversight authority, or for the public at large. In 
many cases, systems are carefully evaluated and tested before they make it to 
the real world. Evidence that is convincing to the public and sufficiently 
nonsensitive to be disclosed widely is the most effective and desirable for 
ensuring accountability. 

In this Part, we examine how computer scientists think about software 
assurance, how software is built and tested in the software industry, and what 
tools are available to get assurances about an individual piece of software or 
a large computer system. Thus, this Part provides a brief and accessible map 
of key concepts and offers some insight into how computer scientists think 
about and approach these challenges. 


A. Assessing Computer Systems 


In general, a computer program is something that takes a set of inputs and 
produces a set of outputs. All too often, programs fail to work as their authors 
intended because the programs have bugs or make assumptions about the 
input data that are not always true. Programmers often structure or design 
programs with an eye toward evaluation and testing in order to avoid or 
minimize these pitfalls.19 Many respected and popular approaches to software 
engineering are based on the idea that code should be written in ways that 
make it easier to analyze.20 For example, the programmer can: 

e Organize the code into modules that can be evaluated separately and 

then combined.?1 

e Test these modules for proper functionality both individually and in 

groups, possibly even testing the entire computer system end-to- 
end. Such testing generally involves writing test cases, or expected 
scenarios in which each module will run, and may involve running 


computer can omit feedback, allowing the computer to request configurations of actuators that are 
beyond their tolerances, This is a problem especially in the design of robotic arms and fly-by-wire 
systems for aircraft. 

19 See ANDREW HUNT & DAVID THOMAS, THE PRAGMATIC PROGRAMMER: FROM 
JOURNEYMAN TO MASTER 196 (2000) (describing a “[cJulture of [t]esting” in which programmers 
should plan on testing since a “little forethought can go a long way toward minimizing maintenance 
costs and help-desk calls”); see also id. at 41 (advocating for an “orthogonally designed and implemented 
system” because it “is easier to test”). 

20 In particular, Test Driven Development (TDD) is a software engineering methodology 
practiced by many major software companies. For a general description of how TDD integrates 
automated testing into software design, see KENT BECK, TEST-DRIVEN DEVELOPMENT: BY 
EXAMPLE (2003). 

21 See HUNT & THOMAS, supra note 19, at 34-43. 
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test cases each time the software is changed to avoid introducing new 
bugs or taking away functionality unintentionally.22 

e Annotate the code with assertions, simple statements about the code 
that describe error conditions under which the program should crash 
immediately. Assertions are intended to be true if the program is 
running as expected. They become false when something is amiss 
and cause the program to crash (with an error message) rather than 
continuing in an errant state. In this way, assertions are a special kind 
of program error—a point at which a piece of software considers its 
internal state and its environment, and stops if these do not match 
what had been assumed by the program’s author.23 

e Provide a detailed description specifying the program’s behavior 
along with a machine-checkable proof that the code satisfies this 
specification. This differs from using assertions in that the proof 
guarantees ahead of time that the program will work as intended in 
all cases (or, equivalently, that an assertion of the facts covered by the 
proof will never fail to be true). When feasible, this approach is the 
most helpful thing a programmer can do to facilitate testing because 
it can provide real proof (rather than just circumstantial evidence or 
evidence linked to a particular point in a program’s execution, as with 
assertions) that the whole program operates as expected.24 

These techniques are illustrative examples from a larger toolbox. Testing 

and verification of software and the development of tools to facilitate it 
comprise a rich and active subfield of computer science research. A thriving 


22 See STEVE MCCONNELL, CODE COMPLETE: A PRACTICAL HANDBOOK OF SOFTWARE 
CONSTRUCTION 528 (2d ed. 2004) (explaining regression testing after each change in code); see also 
id. at 499-533 (discussing testing and test cases). 

23 This technique derives from 1 HERMAN H. GOLDSTINE & JOHN VON NEUMANN, 
PLANNING AND CODING OF PROBLEMS FOR AN ELECTRONIC COMPUTING INSTRUMENT 
(1947), but it is now a widely used technique. For a historical perspective, see Lori A. Clarke & 
David S. Rosenblum, 4 Historical Perspective on Runtime Assertion Checking in Software Development, 
31 ACM SIGSOFT SOFTWARE ENGINEERING NOTES 25 (2006). 

24 A simple example is a technique called model checking, which is usually applied to computer 
hardware designs, in which the property desired and the hardware or program are represented as 
logical formulae and an automated tool performs an exhaustive search (i.e., tries all possible inputs) 
to check whether those formulae are inconsistent. See generally EDMUND M. CLARKE, JR. ET AL., 
MODEL CHECKING (1999). An even simpler example comes from the concept of types in 
programming languages, which associate the data values on which the program operates into 
descriptive classes and provide rules for how those classes should interact. For example, it should 
not be possible to add mathematically a number like “42” to a string of text like “Hello, World!” 
Because both kinds of data are represented inside the computer as bits and bytes, without a type 
system, the computer would be free to try executing this nonsensical behavior, which might lead to 
bugs. Type systems can help programmers avoid mistakes and express extremely complex relationships 
among the data processed by the program. For a more thorough explanation of type systems and model 
checking, see BENJAMIN C. PIERCE, TYPES AND PROGRAMMING LANGUAGES (2002). 
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industry builds tools to assist in the development of software, and there is a 
constant debate about the practice of software engineering in the technology 
industry more broadly. Below, we give a brief taxonomy of this area, as well 
as some examples of tools, expectable results, and limitations of each category. 
We conclude this Section by describing why testing code after it has been 
written, however extensively, cannot provide true assurance of how the 
system works, because any analysis of an existing computer program is 
inherently and fundamentally incomplete. This incompleteness implies that 
observers can never be certain that a computer system has a desired property 
unless that system has been designed to guarantee that property. 

When technologists evaluate computer systems, they attempt to establish 
invariants, or facts about a program's behavior that are always true regardless 
of a program’s internal state or the input data the program receives.25 
Invariants can cover details as small as the behavior of a single line of code 
but can also express complex properties of entire programs, such as which 
users have access to which data or under what conditions the program can 
crash. By structuring code modules and programs in a way that makes it easy 
to establish simple invariants, it is possible to build an entire computer system 
for which important desirable invariants can be proved.26 Together, the set of 
invariants that a program should have make up its specification.27 


25 See HUNT & THOMAS, supra note 19, at 110; C.A.R. Hoare, An Axiomatic Basis for Computer 
Programming, COMM. ACM, Oct. 1969, at 576, 576, 577-80 (providing a foundational examination 
of how to prove properties of a program). 

26 See Hoare, supra note 25, at 576-80. 

27 Specifications can be formal and written in a specification language, in which case they are 
rather like computer programs unto themselves. For example, the early models of core internet 
technology were written in a language called LOTOS, built for that purpose, See Tommaso 
Bolognesi & Ed Brinksma, Introduction to the ISO Specification Language LOTOS, 14 COMPUTER 
NETWORKS & ISDN SYSTEMS 25, 25 (1987) (“LOTOS is a specification language that has been 
specifically developed for the formal description of the OSI (Open Systems Interconnection) 
architecture .. . .”), Other common specification languages in practical use include Z and UML. It 
is even possible to build an executable program by compiling such a specification into a programming 
language or directly into machine code, an area of computer science research known as program 
synthesis. See Zohar Manna & Richard Waldinger, 4 Deductive Approach to Program Synthesis, 2 ACM 
TRANSACTIONS ON PROGRAMMING LANGUAGES & SYSTEMS 90, go (1980) (“Program synthesis 
is the systematic derivation of a program from a given specification.”). Research has shown that when 
the language in which a program or specification is written more closely matches a human-readable 
description of the program's design goals, programs are written with fewer bugs. See Michael C. 
McFarland et al., The High-Level Synthesis of Digital Systems, 78 PROC. IEEE 301 (1990) 
(summarizing early “high-level language” oriented program synthesis); see also MCCONNELL, supra 
note 22, at 457 (arguing that reducing the complexity of code so that it is more comprehensible to 
humans increases reliability and reduces errors). Specifications can also be informal and take the 
form of anything from a mental model of a system in the mind of a programmer to a detailed written 
document describing all goals and use cases for a system. The world of industrial software 
development is full of paradigms and best practices for producing specifications and building code 
that meets them. 
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Software code is, ultimately, a rigid and exact description of itself: the 
code both describes and causes the computer’s behavior when it runs.28 In 
contrast, public policies and laws are characteristically imprecise, often 
deliberately so.29 Thus, even when a well-designed piece of software does 
assure certain properties, there will always remain some room to debate 
whether those assurances match the requirements of public policy. The 
methods described in this Article are designed to forward, rather than to 
foreclose, debates about what laws mean and how they ought to work. Our 
approach aims to empower the policy process by empowering the policymaker’s 
tools for dealing with ambiguity and lack of precision, namely review and 
oversight. We wish to show that software does work as described, allowing a 
reviewer to determine precisely which properties of the software created a 
particular rule enforced for a particular decision. Further, if a precise 
specification of a policy does exist, we wish to show that the software which 
claims to implement that policy in fact does so. 

The specification of a system is a critical question for assessment, and 
system implementers should be prepared to describe the invariants that their 
system provides. Verification allows the claims of a system’s implementer to 
constitute evidence that the software in question in fact satisfies those 
claims.30 Without strong evidence of a computer systems correctness, even 
the author of that system cannot reliably claim that it will behave according 
to a desired policy, and no policymaker or overseer should believe such a 
claim. For example, a medical radiation device with a software control module 
was approved for use on patients based on the manufacturer’s claims, but a 
subtle bug in the software allowed it to administer unsafe levels of radiation, 
which resulted in six accidents and three deaths.31 

Computer scientists evaluate programs using two testing methodologies: 
(1) static methods, which look at the code without running the program; and 
(2) dynamic methods, which run the program and assess the outputs for 


28 See DAVID A. PATTERSON & JOHN L. HENNESSY, COMPUTER ORGANIZATION AND 
DESIGN: THE HARDWARE/SOFTWARE INTERFACE 13-16 (sth ed. 2014) (describing the translation 
of software code into instructions that can be executed by hardware). 

29 See, e.g., Joseph A. Grundfest & A.C. Pritchard, Statutes with Multiple Personality Disorders: The 
Value of Ambiguity in Statutory Design and Interpretation, 54 STAN. L. REV. 627, 628 (2002) (explaining 
how legislators often obscure the meaning of a statute to allow for multiple interpretations). 

30 See CARLO GHEZZI ET AL., FUNDAMENTALS OF SOFTWARE ENGINEERING 269-73 (2d 
ed. 2002). 

31 The commission reviewing the accidents determined that overconfidence on the part of 
engineers and operators led to both a failure to prevent the problem in the first place and a failure 
to recognize it as a problem even after multiple accidents had occurred. For an overview, see NANCY 
LEVESON, Medical Devices: The Therac-25, in SAFEWARE: SYSTEM SAFETY AND COMPUTERS app. 
(1995), an update of the earlier article, Nancy G. Leveson & Clark S. Turner, An Investigation of the 
Therac-25 Accidents, COMPUTER, July 1993, at 18. 
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particular inputs or the state of the program as it is running.32 Dynamic 
methods can be divided into (a) observational methods in which an analyst can 
see how the program runs in the field with its natural inputs; and (b) testing 
methods, which are more powerful, where an analyst chooses inputs and 
submits them to the program.33 


1. Static Analysis: Review from Source Code Alone 


Reading source code allows an analyst to learn a great deal about how a 
program works, but it has some major limitations. Code can be complicated 
or obfuscated, and even expert analysis often misses eventual problems with 
the behavior of the program. For example, the Heartbleed security flaw was 
a potentially catastrophic vulnerability for most internet users that was caused 
by a common programming error—but that error made it through an open 
source vetting process and then sat unnoticed for two years, even though 
anyone was free to read and analyze the code during that time.34 While there 
exist automated tools for discovering bugs in source code, even best-of-breed 
commercial solutions designed to discover exactly this class of error did not 
find the Heartbleed bug because its structure was subtly different from what 
automated tools had been designed to recognize.35 This experience underscores 
how difficult it can be to find small and simple mistakes. More complex errors 
evade scrutiny even more easily. 

Further, static methods, on their own, say nothing about how a program 
interacts with its environment.36 A program that examines any sort of 
external data, even the time of day, may have different behavior when run in 
different contexts. For example, it has long been programming practice to use 
the time of day as the starting value for a chaotic function designed to produce 
random numbers in programs that do statistical sampling.37 Such programs 
naturally choose a different sample of data based on the time of day when 


32 See GARY MCGRAW, SOFTWARE SECURITY: BUILDING SECURITY IN 179 (2006) (“Static 
analysis tools can vet software code, either in source or binary form, in an attempt to identify 
common implementation-level bugs such as buffer overflows. Dynamic analysis tools can observe a 
system as it executes.”). 

33 See id. (“Dynamic analysis tools . . . can submit malformed, malicious, and random data to a 
system's entry points in an attempt to uncover faults—a process commonly referred to as fuzzing.”). 

34 Edward W. Felten & Joshua A. Kroll, Heartbleed Shows Government Must Lead on Internet 
Security, SCI. AM. (Apr. 16, 2014), http://www.scientificamerican.com/article/heartbleed-shows- 
government-must-lead-on-internet-security [https://perma.cc/QLN4-TUQM]. 

35 Id. 

36 See MCGRAW, supra note 32, at 201 (describing software security problems related to 
malicious user inputs, register settings, environment variables, file contents, network configuration, 
and other outside factors). 

37 This practice dates back at least as far as the 1989 standard for the C programming language, 
ANSI X3.159-1989. 
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they were started, meaning that their output cannot be reproduced unless the 
time is explicitly represented as an input to the program. 

Depending on the technology used to implement a program, static 
analysis might lead to incomplete or incorrect conclusions simply because it 
fails to consider the dependencies—that is, the other software that a given 
program needs in order to operate correctly.38 For some technologies, the 
same line of code can have radically different meanings based on the version 
of even a single dependency.39 Because of this, it is necessary for static 
analysis to cover a large portion of any system, and to include at least some 
dynamic information about how a program will be run. 

Within limits, static methods can be very useful in establishing facts about 
a program, such as the nature of the data it takes in, the kind of output it can 
produce, the general shape of the program, and the technologies involved in 
the program’s implementation.40 In particular, static analysis can reveal the 
kinds of inputs that might cause the program to behave in particular ways.4! 
Analysts can use this insight to test the program on different types of inputs. 
Advanced analysis can, in some cases, determine aspects of a program’s 
behavior and establish program invariants, or facts about the program’s 


38 See, e.g., Managing Software Dependencies, GOV.UK SERVICE MANUAL, https://www.gov.uk/ 
service-manual/technology/managing-software-dependencies [https://perma.cc/3BA8-CXED] (“Most 
digital services will rely on some third party code from other software to work properly. This is 
called a dependency.”). 

39 For example, it is common to reuse “library” code, which provides generic functionality and 
can be shared across many programs. GHEZZI ET AL., supra note 30. Library functions can be very 
different from version to version, meaning that running a program with a different version of the 
same library can radically change its behavior. It can even change programs that fail to run at all into 
running, working programs. Because Microsoft Windows refers to its system libraries as “Dynamic 
Linked Libraries,” developers often call this “DLL Hell.” Rick Anderson, The End of DLL Hell, 
MICROSOFT (Jan. 11, 2000), https://web.archive.org/web/20081022192553/http://msdn.microsoft.com/ 
en-us/library/ms811694.aspx [https://perma.cc/ WNQ2-AKWg]. Further, in some programming 
languages, such as PHP, the meaning of certain statements is configurable. See The Configuration File, 
PHP, http://php.net/manual/en/configuration. file.php) [https://perma.cc/g LFC-62D9] (describing 
configuration of PHP). 

40 See MCGRAW, supra note 32, at 106, 109 (describing both the benefits and limitations of 
static code analysis). 

41 In programming languages, the most basic structure for expressing behavior that depends 
on a value is a conditional statement, often written as if X then Y else Z. A conditional statement will 
execute certain code (Y) if the condition (X) is met and different code (Z) if the condition is not 
met. Static analysis can reveal where a program has conditional logic, even if it may not always be 
able to determine which branch of the conditional logic will actually be executed. For example, static 
analysis of conditional logic might show an analyst that a program behaves one way for inputs less 
than a threshold and another way otherwise, or that it behaves differently in some particular special 
case. Generalizing this analysis can allow analysts to break the inputs of a program into classes and 
evaluate how the program behaves on each class. For an overview of logical constructs in computer 
programs, see HAROLD ABELSON ET AL., STRUCTURE AND INTERPRETATION OF COMPUTER 
PROGRAMS (2d ed. 1996), which describes the elements of programs and how they can be combined 
and manipulated, and explains conditional expressions and predicates in section 1.1.6. 
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behavior which are true regardless of what input data the program receives.42 
Programs that are specially designed to take advantage of more advanced 
analysis techniques can enable an analyst to use static methods to prove 
formally complex invariants about the program’s behavior.4 On the simplest 
level, some programming languages are designed to prevent certain classes of 
mistakes. For example, some are designed in such a way that it is impossible 
to make the mistake that caused the Heartbleed bug.44 These techniques have 
also been deployed in the aviation industry, for example, to ensure that the 
software that provides guidance functionality on rockets, airplanes, satellites, 
and scientific probes does not ever crash, as software failures have caused the 
loss of several vehicles in the past.45 More advanced versions of these techniques 
may eventually lead to strong invariants being much more commonly and less 
expensively used in a wider range of applications. 

Transparency advocates often claim that by reviewing a program's disclosed 
source code, an analyst will be able to determine how a program behaves.46 
Indeed, the very idea that transparency allows outsiders to understand how a 
system functions is predicated on the usefulness of static analysis. But this 
claim is belied by the extraordinary difficulty of identifying even genuinely 
malicious code (“malware”), a task which has spawned a multibillion-dollar 


42 See Hoare, supra note 25, at 576-80. 

43 See generally Vijay D’Silva et al., 4 Survey of Automated Techniques for Formal Software 
Verification, 27 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN INTEGRATED CIRCUITS & 
SYSTEMS 1165, 1165 (2008) (“Formal verification tools can provide a guarantee that a design is free 
of specific flaws. This paper surveys algorithms that perform automatic static analysis of software to 
detect programming errors or prove their absence.”). 

44 Since Heartbleed was caused by improper access to the program’s main memory, see Felten 
& Kroll, supra note 34, computer scientists refer to the property that a program has no such improper 
access as “memory safety.” For a discussion of the formal meaning of software safety, see PIERCE, 
supra note 24, at 95-80. For an approachable description of possible memory safety issues in software, 
see Erik Poll, Lecture Notes on Language-Based Security ch. 3, https://www.cs.ru.nl/E.Poll/papers/ 
language_based_security.pdf [https://perma.cc/Y]23-6PG6]. Several modern programming languages 
are memory safe, including some, such as Java, that are widely used in industrial software 
development. However, while any system could be written in a memory safe language, developers 
often choose memory unsafe languages for performance and other reasons. 

45 Both the Ariane 5 and Mars Polar Lander crashed due to software failures. See J.L. LIONS, 
CHAIRMAN, ARIANE 501 INQUIRY BD., ARIANE 5: FLIGHT 501 FAILURE (1996); James Gleick, Little 
Bug, Big Bang, N.Y. TIMES MAG. (Dec. 1, 1996), http://www.nytimes.com/1996/12/01/magazine/ 
little-bug-big-bang.html [https://perma.cc/D4JE-V7K2]; see also ARDEN ALBEE ET AL., JPL 
SPECIAL REVIEW BD., REPORT ON THE LOSS OF THE MARS POLAR LANDER AND DEEP SPACE 
2 MISSIONS (2000), http://spaceflight.nasa.gov/spacenews/releases/2000/mpl/mpl_report_1.pdf [https: 
//perma.cc/REgZ-PX6L]. Similarly, a software configuration error caused the crash of an Airbus 
A400M military transport. Sean Gallagher, Airbus Confirms Software Configuration Error Caused Plane 
Crash, ARS TECHNICA (June 1, 2015), http://arstechnica.com/information-technology/2015/06/ 
airbus-confirms-software-configuration-error-caused-plane-crash [https://perma.cc/XgA4-X9CH]. 

46 See FRANK PASQUALE, THE BLACK BOX SOCIETY: THE SECRET ALGORITHMS THAT 
CONTROL MONEY AND INFORMATION 40 (2015). 
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industry based largely on the careful review of code samples collected across 
the internet.47 Of course, under some circumstances, transparency can also 
use dynamic methods such as emulating disclosed code on disclosed input 
data. We discuss transparency further in Part II. 


2. Dynamic Testing: Examining a Program’s Actual Behavior 


By running a program, dynamic testing can provide insights not available 
through static source code review. But again, there are limits. While static 
methods may fail to reveal what a program will do, dynamic methods are 
limited by the finite number of inputs that can be tested or outputs that can 
be observed. This is important because decision policies tend to have many 
more possible inputs than a dynamic analysis can observe or test.48 Dynamic 
methods, including structured auditing of possible inputs, can explore only a 
small subset of those potential inputs.49 Therefore, no amount of dynamic 
testing can make an observer certain that he or she knows what the computer 
would do in some other situation that has yet to be tested.50 

Dynamic testing can be divided into “black-box testing,” which considers 
only the inputs and outputs of a system or component, and “white-box 


47 Malware analysis can also be dynamic. A common approach is to run the code under 
examination inside an emulator and then examine whether or not it attempts to modify security- 
restricted portions of the system's configuration. For an overview, see Manuel Egele et al., 4 Survey on 
Automated Dynamic Malware-Analysis Techniques and Tools, 44 ACM COMPUTING SURVEYS 6 (2012). 

48 Computer scientists call this problem “Combinatorial Explosion.” It is a fundamental 
problem in computing affecting all but the very simplest programs. Edward Tsang, Combinational 
Explosion, U. ESSEX (May 12, 2005), http://cswww.essex.ac.uk/CS P/ComputationalFinanceTeaching/ 
CombinatorialExplosion.html [https://perma.cc/R7KE-4BJD]. 

49 Even auditing techniques that involve significant automation may not be able to cover the 
full range of possible input data if that range cannot be limited in advance to a small enough size to 
be searched effectively. For programmers testing their own software, achieving complete coverage 
of a program’s behavior by testing alone is considered impossible. Indeed, if testing for the correct 
behavior of a program were possible at a modest cost, then there would be no bugs in modern 
software. For a formal version of this argument, see H.G. Rice, Classes of Recursively Enumerable Sets 
and Their Decision Problems, 74 TRANSACTIONS AM. MATHEMATICAL SOC’Y 358 (1953). 

50 Computer security experts often worry about so-called “back doors,’ which are unnoticed 
modifications to software that cause it to behave in unexpected, malicious ways when presented with 
certain special inputs known only to an attacker. There are even annual contests in which the organizers 
“propose a challenge to coders to solve a simple data processing problem, but with covert malicious 
behavior. Examples include miscounting votes, shaving money from financial transactions, or leaking 
information to an eavesdropper. The main goal, however, is to write source code that easily passes visual 
inspection by other programmers.” ‘THE UNDERHANDED C CONTEST, http://www.underhanded-c. 
org/_page_id_2.html [https://perma.cc/82N4-FBDP]. Back doors have been discovered sitting 
undetected for many years in commercial, security-focused infrastructure products subject to significant 
expert review, including the Juniper NetScreen line of devices. See Matthew Green, On the Juniper 
Backdoor, FEW THOUGHTS ON CRYPTOGRAPHIC ENGINEERING (Dec. 22, 2015), http://blog.crypto 
graphyengineering.com/2015/12/22/on-juniper-backdoor [https://perma.cc/M7S8-SCM4] (describing the 
unauthorized code that created a security vulnerability in the Juniper devices). 
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testing,” in which the structure of the system’s internals is used to design test 
cases. Intuitively, white-box evaluation is more powerful, since any test that 
can be performed in a black-box setting can also be performed in a white-box 
setting, but white-box evaluation can suggest more robust test cases by 
showing an analyst when multiple tests will yield the same behavior or what 
inputs are likely to trigger differences in the output.5! White-box analysis also 
helps the developers and operators of a system determine how to monitor its 
operation so that deviations from expected behavior (e.g., unforeseen bugs, 
security compromise, abuse, and other unexpected behavior) can be detected 
and remedied.%2 

Even structured “audits” of software systems, in which systems are 
provided with related inputs and analyzed for differential behavior, cannot 
provide complete coverage of a program’s behavior for the same reason: this 
methodology explains little about what happens to inputs which have not 
been tested, even those that differ very slightly.53 Additionally, auditing that 
treats a system as a black box tells an analyst very little about why differential 
behavior was observed. A computer program could treat two inputs very 
differently because it has been explicitly designed to use special case logic for 
one or both, because those inputs naturally fall into different decision 
categories, or because the decision rule in use is very sensitive to small 
changes in its input. 

One extremely straightforward and very commonly used form of dynamic 
program review comes from the practice of logging, or recording certain 
program actions in a file either immediately before or immediately after they 
have taken place.54 Analysis of log messages is among the easiest and is 
perhaps the most common type of functional review performed on most 
software programs. However, analyzing program logs requires that programs 
be written to log when they perform events which might be interesting for 
analysis (and that they log enough information about those events to actually 
perform the analysis in question).55 And because logs are just like other files 
on a computer, they can easily be modified and rewritten to contain a 
sequence of events that bears no relation to what a system's software actually 


51 See MCGRAW, supra note 32, at 106, 109 (describing both the benefits and limitations of 
static code analysis). 

52 See SLAWEK LIGUS, EFFECTIVE MONITORING AND ALERTING 1-2 (2012) (describing how 
to perform monitoring effectively, as opposed to verifying a system's behavior through testing alone). 

53 See infra note 58 and accompanying text. 

54 Logging is now sufficiently common to be a basic feature of most programming languages. For 
an overview of early uses, see Ronald E. Rice & Christine L. Borgman, The Use of Computer-Monitored 
Data in Information Science and Communication Research, 34 J. AM. SOC’Y INFO. SCI. 247 (1983). 

55 See, e.g., Bernard J. Jansen, Search Log Analysis: What It Is, What’s Been Done, How to Do It, 28 
LIBR. & INFO. SCI. RES. 407 (2006). 
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did. Because of this, audit logs meant to record sensitive actions requiring 
reliable review are generally access controlled or sent to special restricted 
remote systems dedicated to receiving logging data.56 


3. The Fundamental Limit of Testing: Noncomputability 


Testing of any kind is, however, a fundamentally limited approach to 
determining whether any fact about a computer system is true or untrue. 
There are some surprising limitations to the ability to evaluate code statically 
or dynamically. Counterintuitively, the power of computers is generally 
limited by a concept that computer scientists call noncomputability.57 In short, 
certain types of problems cannot be solved by any computer program in any 
finite amount of time. There are many examples of noncomputable problems, 
but the most famous is Alan Turing’s “Halting Problem,” which asks whether 
a given program will finish running (“halt”) and return an answer on a given 
input or will run forever on that input. No algorithm can solve this problem 
for every program and every input.58 As a corollary, no testing regime can 
establish any property for all possible programs, since no regime can even 
determine whether all programs will actually terminate.59 A related theorem, 
proposed by Rice, strongly limits the theoretical effectiveness of testing, 
saying that for any nontrivial property of a program’s behavior, no algorithm 
can always establish whether a program under analysis has that property.60 
Any such algorithm must get some cases wrong even if the algorithm can do 
both static and dynamic analyses of the program.! However, testing can be 
very useful in establishing certain specific invariants on restricted classes of 
programs, and can be made much more useful when programs are designed 


56 A common feature of security breaches of computer systems is that attackers will rewrite 
logs to prevent investigation into how the attack was carried out or who did it. See ERIC COLE ET 
AL., NETWORK SECURITY BIBLE 198 (2005) (noting that the “early stages of an attack often deal 
with deleting and disabling logging”). Modifying logs in this way can even allow attackers to avoid 
losing access to a compromised system once the compromise has been detected, since it obscures 
what steps must be taken to remediate the intrusion. See generally id. (describing how security breaches 
happen, how they are investigated, and how attackers try to cover the traces of their activity). 

57 See 2 MICHAEL SIPSER, INTRODUCTION TO THE THEORY OF COMPUTATION 201 (2006) 
(noting the existence of “computationally unsolvable problems”). 

58 See A.M. Turing, On Computable Numbers, with an Application to the Entscheidungsproblem, 42 
PROC. LONDON MATHEMATICAL SOC’Y 230, 259-63 (1937) (proving that the Hilbert 
Entscheidungsproblem has no solution); see also A.M. Turing, On Computable Numbers, with an 
Application to the Entscheidungsproblem. A Correction, 43 PROC. LONDON MATHEMATICAL SOC’Y 
544, 544-46 (1937) (providing a correction to the original proof). 

59 To see why this is so, imagine writing a new program which halts if it decides that the 
program it is testing has a certain property, and which runs forever otherwise. For a more detailed 
version of this argument, see SIPSER, supra note 57, at 219, 241, 243 (discussing Rice’s Theorem). 

60 Rice, supra note 49. 

61 Id. 
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to facilitate the use of testing to establish those invariants. That is, while 
testing is not guaranteed to work in general, it can often be useful in specific 
cases, especially when those cases have been designed to facilitate testing. 

While both static and dynamic methods are after-the-fact assessments, as 
they take the computer system and its design as a given, using both approaches 
together is often helpful. If an analyst can establish through static methods 
that a program behaves identically over some class of inputs,62 the analyst can 
test a single input from that class and infer the program behavior for the rest 
of the class. However, not every computer program will be able to be fully 
analyzed, even with such a combination of methods. 


B. The Importance of Randomness 


Randomness is essential to the design of many computer systems, so any 
approach to accountability must grapple with it.’ However, when 
randomness is used, it is easy to lose accountability, since by definition any 
outcome which a randomized process could have produced is at least facially 
consistent with the design of that process.64 Accountability for randomized 


62 This can be done, for example, by noting where in a program’s source code it considers input 
values and changes its behavior. See infra note 67 and accompanying text. 

63 In fact, there is suggestive theoretical evidence that the power of randomness may be 
fundamental: there are problems for which the best known randomized algorithm performs much 
better than the best known deterministic algorithm. For example, the well-studied “multi-armed 
bandit” problem in statistics has seen wide application in the field of machine learning, where 
randomized decisionmaking strategies are provably more efficient than nonrandomized ones. See, 
e.g., J.C. Gittins, Bandit Processes and Dynamic Allocation Indices, 41]. ROYAL STAT. SOC’Y 148, 148 (1979) 
(providing a formal mathematical definition of the multi-armed bandit problem); see also RICHARD S. 
SUTTON & ANDREW G. BARTO, REINFORCEMENT LEARNING 26 (1998) (providing a general 
overview of the usefulness of the multi-armed bandit problem in machine learning applications). 

Even outside of machine learning, there are strong indications in computer science theory that 
certain problems can be solved efficiently only via randomized techniques. Although it is obvious 
that every efficient algorithm also has an efficient randomized version (which is just rewritten to 
take some random bits as input and ignore them), it is conjectured but not known that the converse 
is not true, namely that every efficient randomized algorithm also has a deterministic version that 
solves the same problem with comparable efficiency. For a summary of work in this area, see Leonid 
A. Levin, Randomness and Nondeterminism, 1994 PROC. INT'L CONGRESS MATHEMATICIANS 1418. 
Many important problems, from finding prime numbers (which is necessary for much modern 
cryptography), to estimating the volume of an object (which is useful in computer graphics and 
vision algorithms), to most machine learning, had well-understood randomized algorithms that 
solved them long before they had efficient deterministic solutions (many still do not have any known 
efficient and deterministic algorithms), 

64 For example, a winning lottery ticket with the numbers “1 2 3 4 5” is just as likely to be 
correct as any other ticket, and yet it seems strikingly unlikely. In a similar way, it will always be 
necessary when randomness is involved in a process to ensure that even outcomes that are “correct” 
in the sense that the system could have produced them are also correct in the sense that they fulfill 
the goals which necessitated randomness in the first place (e.g., in a lottery, that the winning ticket 
numbers not be known in advance of their selection and not be influenced by the lottery operators). 


654 University of Pennsylvania Law Review [Vol. 165: 633 


processes must determine why randomness was needed and determine that 
the source of that randomness and its incorporation into the process under 
scrutiny meets those goals. 

The most intuitive benefit of randomness in a decision policy is that it 
helps prevent strategic behavior—i.e., “gaming” of a system. When a tax 
examiner, for example, uses software to choose who is audited, randomization 
makes it impossible for a taxpayer to be sure whether or not he or she will be 
audited. Those who are evading taxes, in particular, face an unknown risk of 
detection, which can be minimized but not eliminated, and do not know 
whether, or when, they should prepare to be audited. Similarly, if additional 
security screening is applied at random to those crossing a checkpoint, or if 
the procedures at the checkpoint are changed at random on a day-to-day basis, 
a smuggler or attacker cannot be as prepared as if the procedures were fully 
known in advance.65 Additionally, studies of the performance of human 
guards have shown that randomization in procedures reduces boredom, 
thereby improving vigilance.66 

The card game of poker illustrates a second benefit: randomness can 
obscure secret information. A good poker player has secret information—how 
good her cards are—that affects how she will bet. By occasionally bluffing, she 
randomizes her behavior and makes it more difficult for opponents to infer 
the quality of her hand. 

In situations where a scarce or limited resource must be apportioned to 
equally deserving recipients such that not all qualified applicants can receive 
a share, randomness can help by fairly apportioning resources to participants 
in a way that cannot be controlled or predicted by those in control of the 
resource. For example, the Diversity Visa Lottery, considered in Part II, is a 
case where a random lottery allocates a scarce resource—a limited number of 
visas to live and work in the United States. Randomness as a source of fairness 
requires two attributes: first, the outcome must not be controlled by the 
systems operator, or else the randomness serves little purpose when 
compared to a model where the system's operator just chooses the winners; 
and second, the outcome must not be predictable, or else the operator of such 
a system could put its favored winners into certain slots or slip them “winning 


65 See, e.g., James Pita et al., Deployed ARMOR Protection: The Application of a Game Theoretic 
Model for Security at the Los Angeles International Airport, 2008 PROC. 7TH INT'L CONF. ON 
AUTONOMOUS AGENTS & MULTIAGENT SYSTEMS: INDUSTRY & APPLICATIONS TRACK 125 
(describing a software system that uses a game-theoretic randomized model to improve the 
efficiency of police and federal air marshal patrols at the Los Angeles International Airport). 

66 See, e.g., RICHARD I. THACKRAY ET AL., FAA CIV. AEROMEDICAL INST., PHYSIOLOGICAL, 
SUBJECTIVE, AND PERFORMANCE CORRELATES OF REPORTED BOREDOM AND MONOTONY 
WHILE PERFORMING A SIMULATED RADAR CONTROL TASK (1975) (discussing the improvement of 
performance through increased unpredictability in procedures). 
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tickets” prior to the systems operation. Further, it is important that the 
random choices made when the system is run be binding upon the system’s 
operator, so that the system cannot be run many times to control the eventual 
output by shopping for a favorable result among many actual runs of the 
system. We explore precisely how to address these issues below. 

Many machine learning systems use randomization as part of their normal 
operation. It turns out that guessing randomly and adjusting the probability 
of each class of output often leads to much better performance than trying to 
determine the absolute best decision at any point.67 

Finally, randomization can give computers more flexibility to perform 
well in unexpected environments. Consider how the Roomba robot is 
programmed to vacuum rooms.68 If rules of motion were hard-coded in the 
software controlling the robot, an unusual furniture configuration might lead 
to the Roomba getting stuck in a corner or under a table or repeatedly following 
the same path without cleaning the rest of the room. Adding in randomized 
motion allows it to escape these patterns and work more effectively without 
the need to code in all possible room configurations. By allowing for unknowns, 
randomized strategies can avoid worst-case outcomes with high probability, 
no matter how unfriendly the environment turns out to be.69 

However, poorly designed randomization can lead to unaccountable 
automated decisions. If a decision depends on a randomly selected value, then 
any outcome consistent with some possible value of the random choice, no 
matter how unlikely, must be considered valid. Concretely, if a decision is 


67 The use of randomization is found throughout artificial intelligence and machine learning. For 
a survey of the field, see STUART RUSSELL & PETER NORVIG, ARTIFICIAL INTELLIGENCE: A 
MODERN APPROACH (1995). Some models naturally depend on randomness. See, e.g., KEVIN B. KORB 
& ANN E. NICHOLSON, BAYESIAN ARTIFICIAL INTELLIGENCE § 1.1 (2004) (describing a model of 
probabilistic reasoning that depends on reckoning with randomness), Other methods simply use 
randomness as an efficient way to explore a large space of possible strategies, in which case it is generally 
necessary to try to build a model many times, evaluate the differences, and report back an estimate of 
confidence in the system's correct construction, See Volodya Vovk et al., Machine-Learning Applications 
of Algorithmic Randomness, 1999 PROC. 16TH INT'L CONF. ON MACHINE LEARNING 444 (describing 
one approach to integrating randomness to improve a machine learning model). 

68 For a description of the Roomba’s movement algorithm, see Ja-Young Sung et al., “My Roomba 
Is Rambo”: Intimate Home Appliances, 2007 PROC. 9TH INTL CONF. ON UBIQUITOUS 
COMPUTING 145, 152, which refers to “the randomness of Roomba’s movement—generated by an 
algorithm designed to promote Roomba’s passage across all sections of the space being cleaned.” 

69 More concretely, one study showed that computer-generated teaching plans customized to 
particular students can be less effective than lesson plans without customization if the software 
model used to tailor lessons to individual performance is trained on large groups that do not capture 
individual-specific patterns. This failure of “big data” methods trained on large groups of students 
to properly capture the quirks of a “small data” situation (such as a classroom-sized group of 
students) can be avoided by adding random deviations from the model’s prediction and tracking the 
results of these deviations. See, e.g., Yun-En Liu et al., Trading Off Scientific Knowledge and User 
Learning with Multi-Armed Bandits, 2014 PROC. 7TH INT'L CONF. ON EDUC. DATA MINING 161 
(observing the introduction of small changes on result prediction). 
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based on the outcome of a coin flip, even if the coin is biased to land heads 
up 99 times out of 100, a result based on a tails up flip cannot be shown to be 
improper, since one out of every 100 results will be derived from the value of 
tails. A corrupt decisionmaker could influence this supposedly random choice, 
picking the value of the coin consistent with its preferred outcome, or could 
flip many coins and then assign the value of each flip to the set of decisions 
he has to make (perhaps by changing the order in which he considers 
decisions) in such a way as to pick the outcomes he desires. Random choices 
generated by a computer system can also be remade by re-running a program 
until the outputs match a preferred outcome. Without designing the computer 
system to demonstrate that this is not happening, it is very hard for a 
decisionmaker to prove that he has not done this. The speed of automated 
decisionmaking only increases this risk; while physical randomization of balls 
in a tumbler can only produce a small number of values per hour of effort, a 
computer can try thousands or millions of outcomes in a matter of minutes. 

Additionally, a randomized process is not easily reproduced. For example, if 
it depends on interaction with its environment (e.g., the operating system on 
which it is running, its human user, or a database with rapidly changing records), 
its behavior may be altered in a nondeterministic way since that environment 
can change between runs.70 One unwieldy solution to this problem is to capture 
all of the environment in which a program runs, so that this environment can be 
replayed by an analyst. However, this solution does not address how to verify all 
of the reasons that randomness might be needed in a process. 


II. DESIGNING COMPUTER SYSTEMS FOR PROCEDURAL REGULARITY 


The first goal in any plan to govern automated decisionmaking should be 
to enable the people overseeing the process—whether they are government 
officials, corporate executives, or members of the public—to know how a 
computer system makes decisions (or, at the very least, that it makes decisions 
based on some rule, even if that rule is not fully disclosed). A baseline 
requirement in most contexts is procedural regularity: each participant will 
know that the same procedure was applied to her and that the procedure was 
not designed in a way that disadvantages her specifically. This baseline 


70 One specific example is a program that chooses a random value based on the time that it has 
been running but takes different amounts of time to run based on what other programs are running 
on the same physical computer system. 

71 For example, a tax auditing risk assessment should not single out individuals either by name 
or by identifying characteristics. If a process added extra weight to filers of a particular postal code, 
gender, and birth month, this could be enough to single out individuals in many cases. See, e.g., Paul 
Ohm, Broken Promises of Privacy: Responding to the Surprising Failure of Anonymization, 57 UCLA L. 
REV, 1701, 1716-27 (2010) (showing that an individual’s identity may be reverse-engineered from a 
small number of data points). 
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requirement draws on the Fourteenth Amendment principle of procedural 
due process. Ever since a seminal nineteenth century case, the Supreme 
Court has articulated that procedural fairness or due process requires rules to 
be generally applicable and not designed for individual cases.72 Similarly, 
federal statutes articulate the requirement for procedural regularity in 
administrative agency actions.73 These principles are also enshrined in the 
Federal Rules of Civil Procedure for civil litigation.74 

In this Part, we will demonstrate that the tools of computer science can 
guarantee important elements of procedural regularity when they are 
incorporated in the initial design of computer systems. Specifically, these 
tools can assure that: 

e The same policy or rule was used to render each decision. 

e The decision policy was fully specified (and this choice of policy was 
recorded reliably) before the particular decision subjects were 
known, reducing the ability to design the process to disadvantage a 
particular individual. 

e Each decision is reproducible from the specified decision policy and 
the inputs for that decision. 

e Ifa decision requires any randomly chosen inputs, those inputs are 
beyond the control of any interested party. 

After describing these properties and showing how they can be 
implemented, we will apply them to a case study—the Diversity Visa Lottery 
at the State Department—where application of these tools could greatly 
improve the legitimacy and fairness of an automated decision procedure. 


A. Transparency and Its Limits 


A naive solution to the problem of verifying procedural regularity is to 
demand transparency of the source code as well as inputs and outputs for the 
relevant decisions; if all of these elements are public, it seems easy to 
determine whether procedural regularity was satisfied. Indeed, full or partial 
transparency can be a helpful tool for governance in many cases,75 and 
transparency has often been suggested as a remedy to accountability issues 


72 See Marchant v. Pa. R.R., 153 U.S. 380, 386 (1894) (holding that the plaintiff had due process 
because “her rights were measured, not by laws made to affect her individually, but by general 
provisions of law applicable to all those in like condition”). 

73 See Administrative Procedure Act, 5 U.S.C. §§ 551-59 (2012) (prescribing exhaustive 
procedural requirements for most levels of federal administrative agency action). 

74 See FED. R. CIV. P. 1 (noting that the rules apply “in all civil actions and proceedings . . . to 
secure the just . . . determination of every action and proceeding” (emphasis added)). 

75 See Danielle Keats Citron & Frank Pasquale, The Scored Society: Due Process for Automated 
Predictions, 89 WASH. L. REV. 1, 8 (2014) (arguing that “transparency of scoring systems is essential”). 
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for computerized systems.7¢ However, transparency alone is not sufficient to 
provide accountability in all cases. 

First and foremost, it is often necessary to keep secret the elements of a 
decision policy, the computer systems that implement it, key inputs, or the 
outcome. Keeping aspects of a decision policy secret can help prevent 
strategic “gaming” of a system. For example, the IRS may look for signs in 
tax returns that are highly correlated with tax evasion based on returns 
previously audited.” But if the public knows exactly which things on a tax 
return are treated as telltale signs of fraud, tax cheats may adjust their 
behavior and the signs may lose their predictive value for the agency. 
Moreover, when the decision being regulated is a commercial one, such as a 
decision to offer credit, a business’s legitimate interest in protecting proprietary 
information or guarding trade secrets like the underwriting formula may be 
incompatible with full transparency. And in many contexts, an automated 
decision may use as inputs, or will create as an output, sensitive or private 
data that should not be broadly shared to protect business interests, privacy, 
or the integrity of law enforcement or investigative methods. In some cases, 
especially with financial or medical data, disclosure may be barred or limited 
by statutes or regulations.78 Finally, in many situations—such as scoring 
consumers for credit or insurance risk—the purpose of the automated 
decision process is to determine something not directly measurable, such as 
the risk of defaulting on credit or claiming a loss on an insurance policy. 
Because these values cannot be measured directly, they are computed from 
proxy variables such as a consumer’s credit history, income, or personal 
attributes. Consumers who understand these actuarial processes would be 
rational in attempting to control the input proxy variables, which in turn 
could render the scoring process less useful in elucidating unmeasurable 
risk.79 Secrecy discourages strategic behavior by participants in the system 
and prevents violations of legal restrictions on disclosure of data. 


76 See 14 C.F.R. § 255.4 (2015) (requiring transparency for airline reservation system display 
information); Frank Pasquale, Beyond Innovation and Competition: The Need for Qualified Transparency 
in Internet Intermediaries, 104 NW. U. L. REV. 105, 160-61 (2010) (suggesting transparency in 
broadband networks to hold carriers accountable for potential favoritism and discrimination). 

77 See Jeff Reeves, IRS Red Flags: How to Avoid a Tax Audit, USA TODAY (Mar. 15, 2015, 12:08 
PM), http://www.usatoday.com/story/money/personalfinance/2014/03/15/irs-tax-audit/5864023 [https:// 
perma.cc/BFW5-DG3a4] (identifying characteristics of tax returns that trigger IRS audit). 

78 See, e.g., 45 C.F.R. § 164.502 (2015) (restricting the disclosure of personally identifiable 
information collected by health care providers). 

79 In particular, consumers are rational to modify proxy variables that control their perceived 
risk when those variables are cheaper or easier to manipulate than the gain obtained via better 
treatment by the decision system. Intuitively, if proxy variables are weak and easy to alter or 
sometimes poorly correlated with the feature being measured (e.g., standardized test scores as a 
measure of student learning), they are more likely to be gamed than features which are highly 
proximate to the value being estimated, or which are difficult or expensive to alter (e.g., annual 
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Second, while transparency allows for the testing strategies described 
earlier (i.e., static and dynamic tests including auditing), those methods are 
often insufficient to verify properties of software systems if these systems 
have not been designed with future evaluation and accountability in mind.80 

Third, for decision processes that involve some element of randomness, 
even full transparency—of the system’s source code, its inputs, its operating 
environment, and its results—does not foreclose the possibility that an 
outcome could be improperly fixed in an undetectable way, as described in 
Section I.C.81 A classic lottery provides an excellent example. A perfectly 
transparent algorithm that uses a random number generator to assign a 
number to each participant and has the participants with the lowest numbers 
win will yield results that cannot be reproduced or verified because the 
random number generator will produce different random numbers each time. 
Reviewing the code alone, or even together with the data fed into it and the 
environment in which it was operated, does not tell us that it was actually used 
to produce a particular result. By design, the process produces unpredictable 
results that are not reproducible. 

Fourth and finally, systems that change over time cannot be fully 
understood through transparency alone. System designers regularly change 
complicated automated decision processes—such as search engine ranking 
methodology,82 spam filter rules,83 intrusion detection system methods,84 or 


income as a measure of creditworthiness in a particular transaction). See generally CATHY O’NEIL, 
WEAPONS OF MATH DESTRUCTION: HOw BIG DATA INCREASES INEQUALITY AND THREATENS 
DEMOCRACY (2016). In economic policymaking, this is sometimes known as Goodhart’s Law, 
popularly formulated as “[w]hen a measure becomes a target, it ceases to be a good measure”; 
Goodhart formulated it more formally as “[a]ny observed statistical regularity will tend to collapse 
once pressure is placed upon it for control purposes.” C.A.E. Goodhart, Problems of Monetary 
Management: The U.K. Experience, in 1 PAPERS IN MONETARY ECONOMICS (1976). Hardt and his 
co-authors have developed adversarial methods for designing automated decision and classification 
systems that remain robust even in the face of gaming, See Moritz Hardt et al., Strategic Classification, 
PROC. 2016 ACM CONF. ON INNOVATIONS THEORETICAL COMPUTER SCI. 111 (discussing 
methods to strengthen classification models). 

80 See supra Part I. 

81 There are ways to incorporate randomness that can be replicated. See infra subsection II.C.4. 

82 For a list of updates to one search engine, see Google: Algorithm Updates, SEARCH ENGINE 
LAND, http://searchengineland.com/library/google/google-algorithm-updates [https://perma.cc/XV 
4Z-AFFo9]. 

83 Many spam filters work by keeping a list of bad terms, email addresses, and computers from 
which to block messages. The most widely used “blacklist” is produced by the organization 
Spamhaus. See SBL Advisory, SPAMHAUS, https://www.spamhaus.org/sbl [https://perma.cc/V9 LN- 
EPKg] (describing the Spamhaus Block List Advisory, “a database of IP addresses from which 
Spamhaus does not recommend the acceptance of electronic mail”). 

84 Intrusion detection systems work in a similar way, using a set of “signatures” to identify bad 
network traffic from attackers. See Karen Kent Frederick, Network Intrusion Detection Signatures, Part 
One, SYMANTEC (Dec. 19, 2001), https://www.symantec.com/connect/articles/network-intrusion- 
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the algorithms that select website ads—in response to strategic behavior by 
participants in the system.85 Computer systems that choose social media posts 
to display to users might respond to user behavior. “Online” machine learning 
systems can update their model for predictions after each decision, incorporating 
each new observation as part of their training data. Even knowing the source 
code and data for such systems is not enough to replicate or predict their 
behavior—we also must know precisely how and when they interacted or will 
interact with their environment. Whether updates to the system are effected 
by human engineers and operators (e.g., a search engine engineer modifies 
the software used to rank pages) or by a machine learning system (e.g., the 
search engine’s software discovers that users more often click the second link 
for a certain query instead of the first, so it reverses their order), transparency 
alone does little to explain either why any particular decision was made or 
how fairly the system operates across bases of users or classes of queries. With 
such systems, there is the added risk that the rule disclosed is obsolete by the 
time it can be analyzed. Online machine learning systems update their 
decision rules after every query, meaning that any disclosure will be obsolete 
as soon as it is made. 


B. Auditing and Its Limits 


Beyond transparency, auditing is another strategy for verifying how a 
computer system works. An audit treats the decision process as a black box 
whose inputs and outputs are visible but whose inner workings are unseen. 86 
The approach has a long history in offline contexts, such as testing for 
discrimination in retail car negotiations.87 For retail car negotiations, the 
transparency of the bargaining process for the purchase of a car is insufficient 
to determine if different prices are offered based on race or gender.88 

In computer science, auditing refers to “an independent evaluation of 
conformance of software products and processes to applicable regulations, 


detection-signatures-part-one [https://perma.cc/QgXY-TQCA] (discussing signatures for network 
intrusion detection systems). 

85 See JURE LESKOVEC ET AL., MINING OF MASSIVE DATA SETS ch. 8 (2d ed. 2014). 

86 See generally Christian Sandvig et al., Auditing Algorithms: Research Methods for Detecting 
Discrimination on Internet Platforms (May 22, 2014), http://www-personal.umich.edu/~csandvig/ 
research/Auditing%20Algorithms%20—%20Sandvig%20—%201CA%202014 %20Data%20and%2o0Discr 
imination%2oPreconference.pdf [https://perma.cc/DSsD-3JYS] (describing algorithm audits and 
reviewing possible audit study designs). 

87 See Ian Ayres, Fair Driving: Gender and Race Discrimination in Retail Car Negotiations, 104 
Harv. L. REV. 817, 818 (1991) (using auditing to determine “[w]hether the process of negotiating 
for a new car disadvantages women and minorities”). 

88 See id. at 827-28 (observing that women and minorities received worse prices than white 
males even when using the same negotiation strategy). 
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standards, guidelines, plans, specifications, and procedures.”89 Auditing is 
intended to reveal whether the appropriate procedures were followed and to 
uncover any tampering with a computer system’s operation. For example, 
there is a substantial body of literature in computer science that addresses 
audits of electronic voting systems, and security experts generally agree that 
proper auditing is necessary but insufficient to assure secure computer-aided 
voting systems. 

Computer scientists, however, have shown that black-box evaluation of 
systems is the least powerful of a set of available methods for understanding 
and verifying system behavior.’ Even for measuring demonstrable properties 
of software systems, such as testing whether a system functions as desired 
without bugs, it is much more powerful to be able to understand the design 
of that system and test it in smaller, simpler pieces.92 Approaches that attempt 
to review system failures simply by looking at how the output responds to 
changes in input are limited by either an inability to attribute a cause to those 
changes or an inability to interpret whether or why a change is significant.% 
Instead, software developers regularly use other, more powerful evaluation 
techniques.94 These include white-box testing (in which a the person doing a 
test can see the system's code and uses that knowledge to more effectively search 
for bugs) and using programming languages that automatically preclude certain 
types of mistakes.95 


89 IEEE Computer Society, IEEE Std 1028 - IEEE Standard for Software Reviews and Audits 
§ 8.1 (Aug. 15, 2008), http://ieeexplore.ieee.org/document/ 4601584 [https://perma.cc/ WLD6-VPUN]. 

90 See Joseph Lorenzo Hall, Election Auditing Bibliography (Feb. 12, 2010), https:// 
josephhall.org/eamath/bib.pdf  [https://perma.cc/L397-AATD] (collecting scholarly literature 
discussing audits in elections). 

91 Specifically, white-box testing, in which an analyst has access to the source code under test, 
is generally considered to be superior; even in cases where the basic testing approach does not make 
use of the structure of the software (e.g., so-called “fuzz testing” where a program is subjected to 
randomly generated input), testing benefits from some access to the structure of programs. See supra 
note 51 and accompanying text. Also, consider the difficulties encountered in one such audit study. 
The authors show a causal relationship between changing sensitive, protected attributes (e.g., 
gender) and the advertisements presented to a user (e.g., advertisements for high-paying jobs). See 
Amit Datta et al., Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and 
Discrimination, 2015 PROC. PRIVACY ENHANCING TECHNOLOGIES 92, 105-06. However, the 
methodology is unable to identify the mechanism of this causation or even whether the results 
discovered will generalize beyond the data seen in the study. Id. at 105. 

92 See supra notes 20-23 and accompanying text on approaches to structuring software. 

93 For example, if the output of a system is an error or other failure such as a crash, it is not 
obvious to an analyst how to modify the output to learn much at all. 

94 See supra Section L.A. 

95 See supra note 24 and accompanying text. 
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C. Technical Tools for Procedural Regularity 


As we demonstrated above, transparency and auditing often do not suffice 
for accountability. In this Section, we introduce computational methods that 
can provide accountability for procedural regularity even when some 
information is kept secret. These methods can be used alongside transparency 
and auditing when appropriate and apply to all computer systems.% 

Our approach harnesses the power of computational methods and does 
not take the design of the computer system as a given. Instead, we explicitly 
advocate for systems to be designed to use methods such as the ones described 
here. Nor do we give up on governance when all or part of a computer system 
must remain secret. We rely on several advanced techniques from computer 
science to enable the governance of secret decision systems: software 
verification, cryptographic commitments, zero-knowledge proofs, and fair 
random choices. Counterintuitively, even when a piece of software or the data 
input to it is secret, these methods can guarantee that the software and inputs 
satisfy the requirements for procedural regularity. They can verify that the 
same decision policy was used for each decision, that the policy was 
determined and recorded before inputs were known, and that the outcomes 
are reproducible. Just because a given computer system or piece of software 
is secret does not mean that nothing about that system can be known. 


1. Software Verification 


Software verification is a set of techniques for proving mathematically that 
software has certain properties, either by analyzing existing code or by building 
software using specialized tools for extracting proved correct invariants. It has 
been a promising field for many decades, and while many benefits are realized 
in research prototypes today, these methods are finding increasing industrial 
adoption, especially in sectors where software is safety- or security-critical and 
in domains with strong liability regimes.97 While the complete verification of 


96 While the methods we propose are general, they can be inefficient for certain applications. 
The cost of providing a certain level of accountability must be considered as part of the design of 
any policy requirement. For more detail, see Kroll, infra note 118. 

97 While software verification has been embraced by the aviation and industrial control sectors 
and for some financial applications (for example, the hedge fund Jane Street regularly touts its use 
of formal software analysis in recruiting materials sent to computer science students), it has yet to 
see much adoption in the critical fields of healthcare and automotive control. See Jean Souyris et al., 
Formal Verification of Avionics Software Products, 2009 PROC. 2ND WORLD CONGRESS ON FORMAL 
METHODS 532 (describing the use of software verification at Airbus); Norbert Volker & Bernd J. 
Kramer, Automated Verification of Function Block-Based Industrial Control Systems, 42 SCI. COMPUTER 
PROGRAMMING 101 (2002). Indeed, researchers have effected compromises of embedded healthcare 
devices such as pacemakers. See, e.g., Daniel Halperin et al., Pacemakers and Implantable Cardiac 
Defibrillators: Software Radio Attacks and Zero-Power Defenses, 2008 IEEE SYMP. ON SECURITY & 
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any program is an expensive undertaking largely reserved to technologists 
versed in this particular area, it is important as a matter of policy to 
understand the options that are available so that costs and benefits can be 
weighed and acted upon. 

Unlike static analysis, which aims to examine already-written code for bugs 
or deviations from its specification, or software testing, which aims to verify 
that software meets a specific set of functional requirements by explicitly 
executing the software in a particular configuration, software verification aims 
to prove invariants about a program mathematically, using logic to reason about 
a program's behavior under all conditions.98 Verified programs come with a 
mathematically checkable proof demonstrating that they have certain invariants, 
rendering testing for those invariants unnecessary since the proof implies that 
such tests will always pass.99 

There are many ways to verify software. For instance, a program can be 
carefully annotated using formal logic to determine its behavior in a precise 
manner, though this can be expensive and will not always yield the desired 


PRIVACY 129, 141 (finding that implantable cardioverter defibrillators are “potentially susceptible to 
malicious attacks that violate the privacy of patient information” and “may experience malicious 
alteration to the integrity of information or state”). News reports also indicate that former Vice 
President Dick Cheney had the remote software update functionality on his pacemaker disabled so 
that updating the software would require surgery, ostensibly in order to prevent remote compromise 
of his life-critical implant. Andrea Peterson, Yes, Terrorists Could Have Hacked Dick Cheney’s Heart, 
WASH. POST (Oct. 21, 2013), https://www.washingtonpost.com/news/the-switch/wp/wp/2013/10/21/ 
yes-terrorists-could-have-hacked-dick-cheneys-heart [https://perma.cc/V Y5S-6XR6]. 

Additionally, researchers have also demonstrated spectacular compromises of automobile 
control systems, including disabling brakes, controlling steering and acceleration, and completely 
cutting engine power. See Karl Koscher et al., Experimental Security Analysis of a Modern Automobile, 
2010 IEEE SYMP. ON SECURITY & PRIVACY 447 (performing early analyses of the security of 
automobile computers); see also Stephen Checkoway et al., Comprehensive Experimental Analyses of 
Automotive Attack Surfaces, 2011 PROC. 20TH USENIX CONF. ON SECURITY 77 (same). Subsequently, 
researchers have demonstrated problems in other models, including luxury models with significant 
telematics capabilities and remote software upgrade capability, showing that active maintenance of 
these software systems does not completely defend against attacks. See, e.g., Jonathan M. Gitlin, 
Man Hacks Tesla Firmware, Finds New Model, Has Car Remotely Downgraded, ARS TECHNICA (Mar. 
8, 2016, 11:36 AM), http://arstechnica.com/cars/2016/03/man-hacks-tesla-firmware-finds-new-model- 
has-car-remotely-downgraded [https://perma.cc/RgCs5-9RTY] (describing an incident where a Tesla 
car model was hacked despite frequent software updates), Problems with spontaneous acceleration 
in many Toyota vehicles were later traced to software issues. See Phil Koopman, A Case Study of 
Toyota Unintended Acceleration and Software Safety (Sept. 18, 2014), https://users.ece.cmu.edu/ 
~koopman/pubs/koopmani14_toyota_ua_slides.pdf [https://perma.cc/VPgT-VYMF] (presenting a 
detailed analysis of the issue). And of course, Volkswagen designed its engine control software to 
defeat an emissions testing regime. For a complete timeline of the Environmental Protection 
Agency’s actions on this matter, see Volkswagen Light Duty Diesel Vehicle Violations for Model Years 
2009-2016, EPA.GOV, https://www.epa.gov/vw [https://perma.cc/C83U-UZLG] (last updated 
Nov. 7, 2016). 

98 See supra Section L.A. 

99 See supra notes 25-27 and accompanying text. 
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invariants;100 a program can be certified by another program that translates it 
to a form guaranteed to have the desired property;!0t a program can be 
exhaustively tested for all possible inputs to ensure that an invariant is never 
violated;102 or a program can be built using tools that allow for the careful 
specification of invariants (and proofs of those invariants).103 Researchers 
have even verified entire operating systems using these techniques.104 
Verification can be communicated to clients in a number of ways: so called 
proof-carrying code comes with a machine-checkable proof of its verified 
invariants, which can be checked by a user just prior to running the 


100 For one of the earliest approaches to representing programs as statements in formal logic, see 
Hoare, supra note 25, at 576-80. While Hoare’s techniques form the basis of many modern methods, 
some methods attempt to build software that is correct by virtue of its construction, rather than 
analyzing software that has already been written. For an overview of different approaches and their 
tradeoffs, see B. BÉRARD ET AL., SYSTEMS AND SOFTWARE VERIFICATION: MODEL-CHECKING 
‘TECHNIQUES AND TOOLS (2001). For a classic reference on how to include these techniques in the 
software engineering process, see GHEZZI ET AL., supra note 30. 

101 These tools are known as “certifying compilers.” The advantage of a certifying compiler is 
that one need only expend effort verifying the certifying compiler itself, not the software being 
compiled, in order to prove that the desired invariant holds for the compiled software. For a 
description of the original concept and a first implementation, see George C. Necula & Peter Lee, 
The Design and Implementation of a Certifying Compiler, 33 ACM SIGPLAN NOTICES 333 (1998). 
There are many examples of certified compiler systems. See, e.g., Joshua A. Kroll et al., Portable 
Software Fault Isolation, 2014 PROC. IEEE 27TH COMPUTER SECURITY FOUND. SYMP. 18 
(describing a certifying software fault isolation compiler built out of CompCert’s certified back end). 

102 This technique, known as “model checking,” could also be described as a form of static 
analysis. Model checking aims to verify an invariant by finding a counterexample (an input to the 
program which makes the invariant untrue and hence not an invariant). If a counterexample can be 
found, the program has a demonstrable bug. If no counterexample can be found, that invariant has 
been verified. See also supra note 24 (discussing model checking). 

103 Several such programming languages exist, though one of the more successful toolkits in 
active research is the proof assistant Coq, which allows for writing complex programs and theorems 
and invariants about those programs in such a way that the proved-correct programs can be 
“extracted” into executable code. For an introduction to Coq, see ADAM CHLIPALA, CERTIFIED 
PROGRAMMING WITH DEPENDENT TYPES: A PRAGMATIC INTRODUCTION TO THE COQ 
PROOF ASSISTANT (2013), and YVES BERTOT & PIERRE CASTERAN, INTERACTIVE THEOREM 
PROVING AND PROGRAM DEVELOPMENT: COQ’ART: THE CALCULUS OF INDUCTIVE 
CONSTRUCTIONS (2004), which describe advanced programming techniques. Several large and 
complex programs have been written in Coq, which demonstrates that it is a robust tool capable of 
supporting nontrivial development tasks and proofs of correctness about those tasks. Perhaps the 
most famous of these was the proof of the “four-color theorem,” which states that any map can be 
drawn using only four colors such that no border on the map uses the same color for the regions on 
both sides of the border. Georges Gonthier, Formal Proof—The Four-Color Theorem, 55 NOTICES 
AMS 1382 (2008). Similar tools include a theorem prover for programs written in ANSI Common 
Lisp 2 and the interactive theorem prover Isabelle. See Lawrence C. Paulson, The Foundation of a 
Generic Theorem Prover, 5 J]. AUTOMATED REASONING 363 (1989) (describing the design and 
implementation of Isabelle). 

104 See Gerwin Klein et al., seL4: Formal Verification of an OS Kernel, 2009 PROC. ACM 
SIGOPS 22ND SYMP. ON OPERATING SYSTEMS PRINCIPLES 207 (documenting the verification of 
the seL4 microkernel). 
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program;105 a user can recompute the analysis used to generate the proof} or 
the truth of the proof can be confirmed by an entity the user trusts, with 
cryptography used to guarantee that the version a user is running matches the 
version that was verified.106 

However, just because a program has been verified or proven correct does 
not mean that it has been vetted at all for correctness or compliance with a 
policy. Verification typically constitutes a proof that the software object in 
use matches its specification, but this analysis says nothing about whether the 
specification is sufficiently detailed, correct, lawful, or socially acceptable, or 
constitutes good policy. Software verification is a rapidly developing field, 
and the costs of building fully verified software will likely drop precipitously 
in the coming decades, leading to wide adoption in the software industry due 
to the benefits of reduced security exposure and the elimination of many 
types of software bugs. 


2. Cryptographic Commitments 


A cryptographic commitment is the digital equivalent of a sealed document 
held by a third party or in a safe place. It is possible to compute a commitment 
for any digital object (e.g., a file, a document, the contents of a search engine’s 
index at a particular time, or any string of bytes). Commitments are a kind of 
promise that binds the committer to a specific value for the object being 
committed to (i.e., the object inside the envelope) such that the object can 
later be revealed and anyone can verify that the commitment corresponds to 
that digital object.107 In this way, as in the envelope analogy, an observer can 
be certain that the object was not changed since the commitment was issued 
and that the committer did indeed know the value of the object at the time the 
commitment was made (e.g., the source code to a program or the contents of a 
document or computer file). Importantly, secure cryptographic commitments 


105 See, e.g., George C. Necula & Peter Lee, Safe Kernel Extensions Without Run-Time Checking, 
1996 PROC. 2ND UNENIX SYMP. ON OPERATING SYS. DESIGN & IMPLEMENTATION 229 
(describing the concept of proof-carrying code and a first implementation with applications to 
operating system security); see also George C. Necula, Proof-Carrying Code Design and 
Implementation., in PROOF AND SYSTEM RELIABILITY 261 (H. Schwichtenberg & R. Steibruggen 
eds., 2002) (giving a detailed overview of the concepts of proof-carrying code and their development 
over time). 

106 This approach would consist of the certifying authority making a cryptographically signed 
statement that it had verified the proof for a binary with a certain cryptographic hash value and the 
distribution of a signed copy of that piece of software. For an overview of code signing systems, see 
Code Signing, CERTIFICATE AUTHORITY SECURITY COUNCIL, https://casecurity.org/wp-content/ 
uploads/2013/10/CASC-Code-Signing. pdf [https://perma.cc/DZU8-TA36]. 

107 See Ariel Hamlin et al., Cryptography for Big Data Security, in BIG DATA: STORAGE, 
SHARING, AND SECURITY 267 (Fei Hu ed., 2016) (describing cryptographic commitments as a 
method of verification). 
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are also hiding, meaning that knowledge of the commitment (or possession of 
the envelope in the analogy) does not confer information about the contents. 
This gives rise to the sealed document analogy: once an object is “inside” the 
sealed envelope, an observer cannot see it nor can anyone change it. However, 
unlike physical envelopes, commitments can be published, transmitted, copied, 
and shared at very low cost and do not need to be guarded to prevent 
tampering. In practice, cryptographic commitments are much smaller than the 
digital objects they represent.108 Because of this, commitments can be used to 
lock in knowledge of a secret (say, an undisclosed decision policy) at a certain 
time (say, by publishing it or sending it to an oversight body) without revealing 
the contents of the secret, while still allowing the secret to be disclosed later 
(e.g., in a court case under a discovery order) and guaranteeing that the secret 
was not changed in the interim (for example, that the decision policy was not 
modified from one that was explicitly discriminatory to one that was neutral).109 

When a commitment is computed from a digital object, the commitment 
also yields an opening key, which can be used to verify the commitment.110 
Importantly, a commitment can only be verified using the precise digital 
object and opening key related to its computation; it is computationally 
implausible for anyone to discover either another digital object or another 
opening key which will allow the commitment to verify properly. In the 
envelope metaphor, this is tantamount to proof that neither the envelope nor 
the document inside the envelope was replaced clandestinely with a different 
envelope or document. Any digital object (e.g., a file, document, or any string 
of bytes) can have a commitment and an opening key such that it is (1) impossible 
to deduce the original object from the commitment alone; (2) possible to 
verify, given the opening key, that the original object corresponds to the 


108 See id. at 267 (noting that commitments can be smaller than the statements to which they 
relate), A typical commitment will be 128 or 256 bytes, regardless of the size of the committed 
object. See INFO. TECH. LAB., NAT’L INST. OF STANDARDS & TECH., FIPS PUB 180-4, SECURE 
HASH STANDARD (2015) (describing the hash algorithms accepted for government computer 
applications, which provide widely used standards in industry). 

109 As a curiosity, we remark that the popular board game Diplomacy is essentially based on 
physical world commitments: each player negotiates a set of moves for the next round of the game, 
but then these moves are written on paper and passed secretly to a game master who stores them in 
an envelope. Once all players have entered their moves, the moves are revealed and taken 
simultaneously. This commitment mechanism allows players to simulate simultaneous moves 
without any risk that a player will fall behind or change their moves in a particular round in response 
to their perception of what another player is doing in that round. However, the commitment 
mechanism alone does not prevent players from entering incorrect or impossible moves, writing 
nonsense on their paper instead of moves, or simply refusing to enter a move at all (the game master, 
however, enforces that all moves placed into the envelope are correct and all players must trust her 
to do this to ensure that the game is not spoiled). Below, in the section on zero-knowledge proofs, 
we describe how techniques from computer science can address the role of the game master purely 
through computation without the need for an entity trusted by all players of the game. 

110 See ODED GOLDREICH, FOUNDATIONS OF CRYPTOGRAPHY — A PRIMER 19 (2005). 
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commitment; and (3) impossible to generate a fake object and fake opening 
key such that using the (real) commitment and the fake opening key will 
reveal the fake object. 

Cryptographic commitments have useful implications for procedural 
regularity in automated decisions. They can be used to ensure that the same 
decision policy was used for each of many decisions. They can ensure that 
rules implemented in software were fully determined at a specific moment in 
time. This means a government agency or other organization can commit to 
the assertions that (1) the particular decision policy was used and (2) the 
particular data were used as input to the decision policy (or that a particular 
outcome from the policy was computed from the input data). The agency can 
prove the assertions by taking its secret source code, the private input data, 
and the private computed decision outcome and computing a commitment 
and opening key (or a separate commitment and opening key for each policy 
version, input, or decision). The company or agency making an automated 
decision would then publish the commitment or commitments publicly and in 
a way that establishes a reliable publication date, perhaps in a venue such as a 
newspaper or the Federal Register. Later, the agency could prove that it had 
the source code, input data, or computed results at the time of commitment by 
revealing the source code and the opening key to an oversight body such as a 
court. This technique assures that the software implementing the decision 
policy was determined and recorded prior to the publication of the 
commitment, which can be useful in demonstrating that neither the software 
nor the decision policy were influenced by later information or events. 

By themselves, however, cryptographic commitments do not prevent the 
committer from lying and generating a fake commitment that it cannot open 
at all or from destroying (or refusing to disclose) the information that allows 
a valid commitment to be opened. In either case, when the time comes to 
reveal the contents of the commitment, it will be demonstrable that the 
committer has misbehaved. However, an observer does not know the nature of 
the misbehavior. The committer may not have a correct opening key (analogous 
to having sealed an unintelligible or irrelevant document in a physical 
envelope) or may want to lie about what was in the original file (analogous to 
discovering that the contents of the envelope may be embarrassing under 
scrutiny of oversight). In either case, an oversight authority might punish the 
committer for lying and assume the worst about the contents of the missing 
file. However, it would be preferable to be able to avoid this scenario 
altogether, which we can do with another tool known as zero-knowledge proofs. 


111 A parallel to this assumption is a spoliation inference, which sanctions a party who 
withholds, tampers, or destroys evidence by assuming that the missing or changed evidence was 
unfavorable to the spoliator. See Fed. R. Civ. P. 37(e)(2)(A) (providing that if electronically stored 
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3. Zero-Knowledge Proofs 


A zero-knowledge proof is a cryptographic tool that allows a decisionmaker, 
as part of a cryptographic commitment, to prove that the decision policy that 
was actually used (or the particular decision reached in a certain case) has a 
certain property, but without having to reveal either how that property is 
known or what the decision policy actually is.112 

For example, consider how money flows in an escrow transaction. 
Traditionally, an escrow agent holds payment until certain conditions are met. 
Once they are, the agent attests to this fact and disburses the money according 
to a predetermined schedule. Zero-knowledge proofs can allow escrow without 
a trusted agent. Suppose that an independent sales contractor wishes to 
certify that she has remitted appropriate taxes from her sales to be paid by a 
counterparty, but without revealing precisely how much she was able to sell 
an item for. Using a zero-knowledge proof, she can demonstrate that sufficient 
taxes were paid without disclosing her sales prices or earnings to a third party. 

Another classic example used in teaching cryptography posits that two 
millionaires are out to lunch and they agree that the richer of them should 
pay the bill. However, neither is willing to disclose the amount of her wealth 
to the other. A zero-knowledge proof allows them both to learn who is 
wealthier (and thus who should pay the restaurant tab) without revealing how 
much either is worth. 

A zero-knowledge proof works with cryptographic commitments to verify 
procedural regularity in the following manner. If a decisionmaker makes a trio 
of commitments, 4, B, and C, where 4 is a commitment to the decision policy, 
B is a commitment to the inputs that were used in a particular case, and C is a 
commitment to the decision actually reached in that case, then zero-knowledge 
proofs let the public verify that 4, B, and C really do correspond to each other. 
In other words, the decisionmaker can prove that, when the committed policy 
A is applied to the committed input data B, the result is the committed 
outcome C. This allows decisionmakers to build audit logs, which can be 
verified by the public to confirm that the decisionmaker applied the 
appropriate policy to the correct input in order to reach the stated outcome, 
all without revealing the decision policy itself and without revealing private 
data that might be included in the input or outcome. 

Later, if the outcome is challenged, a court or other oversight body can 
compel the decisionmaker to reveal the actual policy and input used and can 
verify that it matches the published commitment, effectively providing digital 


information is lost because a party, intending to deprive the other party of the information, failed 
to take reasonable steps to preserve it, the court may “presume that the lost information was 
unfavorable to the party”). 

112 See GOLDREICH, supra note 110, at 16. 
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evidence that the decisionmaker was honest about its announced decision. By 
using a commitment to the same policy in decisions for multiple decision 
subjects, a decisionmaker can demonstrate that it is applying a consistent policy 
across those subjects. Such zero-knowledge proofs can be enhanced to test parts 
of the decision policy, either by exhibiting properties of the input-output 
relation (e.g., that a credit score would have been the same if the subject’s 
gender were reversed) or properties of the policy itself (e.g., that the policy 
only uses certain inputs for certain purposes). 


4. Fair Random Choices 


Where random choices are part of a decisionmaking process, the fairness 
of the randomness used in those computer systems should be verifiable. 
Poorly designed randomization can lead to unaccountable automated 
decisions. The decisionmaker could influence the supposedly random choices 
or could generate many sets of random values and then pick the set that gives 
its preferred outcome. Additionally, a randomized process is not easily 
reproduced. For example, if it depends on interaction with its environment 
(e.g., the operating system on which it is running or its human user), its 
behavior may be altered in a nondeterministic way since that environment 
can change between runs.18 

Automated decision processes must therefore be designed from the 
beginning to allow for oversight of the decisionmaker and to avoid problems 
with unpredictable behavior. To solve this problem, a decisionmaker could 
demonstrate that any unpredictable behavior or random choices in the 
software does not affect the eventual output; for example, a program designed 
to find the top of a hill (i.e., optimize some objective) can start at any 
randomly chosen point and take any arbitrary path upwards and will still 
ultimately return the same maximum value.14 

More often, the random choices made by an automated decision process 
will affect the results. In these cases, the software implementing the decision 


113 One example is a program that chooses a random value based on the time that it has been 
running but takes different amounts of time to run based on other programs that are running on the 
same physical computer system. 

114 In general, this approach will only find the top of some crest, which may or may not be the 
highest point on a hill (for instance, if a mountain has two peaks, one much higher than the other). 
Randomness helps fix this problem, however, since an algorithm can start climbing the hill at many 
different randomly chosen points and verify that they all reach the same highest point. Additionally, 
for many important problems, one can prove that only a certain limited number of optimal (i.e., 
highest or lowest) values exist. That is, if an analyst knows that the hill has only one peak, then 
which path a program takes to the top is irrelevant. For a description of the gradient descent 
approach to optimization and other approaches, see RICHARD O. DUDA ET AL., PATTERN 
CLASSIFICATION 224-27 (2d ed. 2001). 


670 University of Pennsylvania Law Review [Vol. 165: 633 


can always be redesigned to replace the set of random choices made by the 
software with a small, recorded random input (a seed value) from which any 
necessary random values can be computed in a deterministic, pseudorandom 
way. In this way, the decisionmaking process can be replayed so long as the 
seed is known and the randomness of the input is completely captured by the 
randomness of the seed. Using this technique, a decisionmaker would not 
have to generate a new random choice each time a random value is needed by 
a piece of software (such choices can be made by a cryptographic algorithm 
that uses the seed to yield reproducible values), nor know in advance how 
many random choices must be made. This technique allows software that 
makes random choices, such as a lottery, to be made fully reproducible and 
reviewable. Unlike capturing the entire environment, as was discussed above, 
this technique reduces the relevant portion of the environment to a very small 
and manageable value (the seed) while preserving the benefits of using 
randomness in the system. 

If this technique is used, we also must prevent the decisionmaker from 
tampering with the seed value, as it fully determines all random data accessed 
by the program implementing the decision policy. Several methods can aid in 
ensuring the fair choice of seed values. A public procedure can be used to 
select a random value: for example, rolling dice or picking ping pong balls 
from the sort of device used by state lotteries.15 Alternatively, the seed value 
could be provided by a trusted third party, such as the random “beacon” 
operated by the U.S. National Institute of Standards and Technology 
(NIST).116 In addition, it is possible for a set of mutually distrustful parties 


115 Currently known strategies for generating public random values (“randomness beacons”) 
all have advantages as well as disadvantages—dice could be weighted; ping pong balls could be put 
in the freezer and the cold ones picked out of the machine. The National Institute of Standards and 
Technology runs a randomness beacon that has come under scrutiny because of distrust of the 
National Security Agency. To minimize these types of issues, the algorithm designer should pick 
the source of randomness most likely to be trusted by participants, which may vary. The algorithm 
designer could choose to collect many sources of random choices and mix them together to maximize 
the number of participants who will trust the randomness of the chosen seed. However, even 
physical sources of randomness that have not been tampered with have failed to be accountable for 
their goals in unexpected ways; for instance, the 1969 lottery for selecting draftees by birthday was 
later shown to be biased, with a disproportionate number of selectees coming from months early in 
the year. For a detailed overview of the problem and its causes, see Joan R. Rosenblatt & James J. 
Filliben, Randomization and the Draft Lottery, SCIENCE, Jan. 22, 1971, at 306. 

116 Computer science refers to a trusted third-party source of randomness as a “beacon.” The 
best known beacon is operated by the NIST, which publishes new random data every few minutes, 
ostensibly based on the measurement of quantum mechanical randomness via a device maintained 
in a NIST lab, NIST Randomness Beacon, NAT’L INST. STANDARDS & TECH., http://www.nist.gov/ 
itl/csd/ct/nist_beacon.cfm [https://perma.cc/UNT3-6N6P] (last updated Sept. 21, 2016). Recent 
revelations about NIST’s role in allowing the U.S. National Security Agency to undermine the 
security of random number generation techniques standardized by NIST have led to some distrust 
of the NIST beacon, although it may be trustworthy in some applications. NIST standardized the 
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(possibly including decision subjects themselves) to engage in an interaction 
that produces a value that is unpredictable so long as at least one participant 
provided random input.t7 Perhaps the best option is to mix together 
randomness (sometimes called entropy) from many different sources. The 
simplest form of this practice would involve a decision subject entering a short 
random number as part of the input for their decision (e.g., on an application 
form). Then, the decisionmaker would generate a seed value for each decision 
by combining this known-to-the-subject, personal random value with (1) a 
pre-chosen random value to which the decisionmaker publicly committed to 
using far in advance of seeing the personal random value, and (2) a unique 
identifier for the particular decision or decision subject that is difficult to 
change (e.g., the social security number of the participant). In order to foster 
maximum confidence that random choices are not improperly influenced, the 
decisionmaker should derive them using a combination of (1) a random value 
from a trusted third-party; (2) a random value chosen by the decisionmaker 
and possibly kept secret; (3) a participant- or decision-specific identifier that 
cannot be changed or controlled by the decisionmaker, such as a social 
security number, identification number, or other immutable piece of the 
subject’s name or data; and (4) a value chosen by the decisionmaker. Since 
these values are either outside of the decisionmaker’s control or are known, 
fixed, and subsequently verifiable before the inputs to a decision are known, 
using these methods gives assurance that the decisionmaker is not skewing 
the results by controlling the selection of random values.118 


Duel EC Deterministic Random Bit Generator (DUAL-EC) in SP 800-90A in 2007. At that time, 
cryptographers already knew the standard could accommodate a “backdoor,” or secret vulnerability. 
See Dan Shumow & Niels Ferguson, On the Possibility of a Back Door in the NIST SP800-90 Dual 
Ec Prng (2007), http://rump2007.cr-yp.to/15-shumow.pdf [https://perma.cc/VC7G-G23G]. Later, it 
was discovered that the NSA had very likely made use of this mechanism to create a backdoor in the 
standard itself. See Daniel J. Bernstein et al., Dual EC: A Standardized Back Door, in THE NEW 
CODEBREAKERS 256 (2016). Other beacon implementations have been proposed, including beacons 
based on “cryptocurrencies” such as Bitcoin. See, e.g., Joseph Bonneau et al., On Bitcoin as a Public 
Randomness Source, https://eprint.iacr.org/2015/1015.pdf [https://perma.cc/XQ38-F]3H] (outlining a 
specific alternate proposal involving the use of Bitcoin as a source of publicly verifiable randomness). 

117 Computer science has methods to simulate a trusted third party making a random choice. 
These methods require the cooperation of many mutually distrustful parties, such that as long as 
any one party chooses randomly, the overall choice is random. By selecting many participants in this 
process, one can maximize the number of people who will believe that the chosen value is in fact beyond 
undue influence. For an easy-to-follow introduction to these methods, see Manuel Blum, Coin Flipping 
by Telephone: A Protocol for Solving Impossible Problems, 15 ACM SIGACT NEWS 23 (1983). 

118 When the fairness of random choices is key to the accountability of a decision process, great 
care must be taken in determining the source of random seed values, as many very subtle 
accountability problems are possible. For example, by changing the order in which decisions are 
taken, the decisionmaker can effectively “shop” for desirable random values by computing future 
deterministic pseudorandom values and picking the order of decisions based on its preference for 
which decisions receive which random choices. To prevent this, it may also be necessary to require 
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Where random choices are part of a decisionmaking process, the fairness 
of the randomness (i.e., the consistency with the goal for which randomness 
was deployed in a particular system) used in those decisions should be 
verifiable. This can be achieved by relying on a small random seed and 
verifying its source. Once a random seed has been chosen in a satisfactory 
manner, it is still necessary to verify that the seed was in fact used in later 
decisions.119 This can be accomplished by the techniques we describe. 


D. Applying Technical Tools Generally 


Our general strategy in designing systems accountable for their procedural 
regularity is to require systems to create cryptographic commitments as digital 
evidence of their actions. Systems can be designed to publish commitments 
describing what they will do (i.e., a commitment to the decision policy 
enforced by the system, as represented by source code) before they are fielded 
and commitments describing what they actually did (i.e., a commitment to 
the inputs!20 and outputs of any particular decision) after they are fielded. 
Zero-knowledge proofs can be used to ensure that these commitments actually 
correspond to the actions taken by a system.!2! Indeed, it is possible to use 
zero-knowledge proofs to verify, for each decision, that the committed-to 
decision policy applied to the committed-to inputs yields the committed-to 
outputs.122 These zero-knowledge proofs could either be made public or 
provided to the system’s decision subjects along with their results. 

By disclosing commitments instead of source code or inputs and outputs, 
system operators can fully explain what their systems do without actually 
disclosing how those systems work up front. Later, if it becomes necessary to 


that the decisionmaker take decisions in a particular order or that the decisionmaker commit to the 
order in which it will take decisions in advance of the seed being chosen. For a detailed description 
of the problems with randomness “shopping” and post-selection by a decision authority, see Joshua 
Alexander Kroll, Accountable Algorithms (Sept. 2015) (unpublished Ph.D. dissertation, Princeton 
University) (on file with author), 

119 For example, several state lotteries have been defrauded by insiders who were able to control 
what random values the lottery system used to decide winners. Specifically, an employee of the 
Multi-State Lottery Association (MUSL) was convicted of installing software on the system that 
controlled the random drawing and using the information gleaned by the software to purchase 
winning tickets for the association’s “Hot Lotto” game. See Grant Rodgers, Hot Lotto Rigger Sentenced 
to 10 Years, DES MOINES REG. (Sept. 9, 2015, 7:12 PM), http://www.desmoinesregister.com/story/ 
news/crime-and-courts/2015/09/09/convicted-hot-lotto-rigger-sentenced-10-years/71924916  [https:// 
perma.cc/U26A-8VMD] (describing the Iowa lottery fraud sentencing). 

120 Note that, for these commitments to function, systems must also be designed to be fully 
reproducible, capturing all interactions with their environments as explicit inputs that can then be 
contained in published commitments. The use of seed values for randomization, discussed above in 
subsection II.C.4, offers one example of ensuring reproducibility. 

121 The approach here was introduced in Kroll, supra note 118. 

122 Id. 
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review the actions or decision policy of a system during a court case or 
regulatory action, system operators can disclose the contents of their 
commitments (that is, source code, inputs, and outputs), possibly under a 
protective regime. If it is possible to disclose these values publicly, then system 
operators may also choose (or be required) to do so. However, whether these 
data are disclosed or not, the published commitments and zero-knowledge 
proofs allow overseers and the public at large to verify that the decisions of 
some authority actually correspond to a specific predetermined policy rather 
than the arbitrary whim of a decisionmaker. Further, by observing that all 
decisions arise from the same policy, anyone reviewing these commitments 
can be certain that the policy was used for all decisions simply by checking 
that the commitments to the decision policy are consistent across decisions. 

By requiring commitments to be published far in advance of any decision, 
it is possible to ensure that the particulars of a decision policy were chosen 
independently of the factors in the decisions it would render. For example, a 
decision policy that selects which individuals will receive a tax audit should 
be based on the risk of tax evasion, which in turn can be inferred by properties 
of the tax return itself. However, a corrupt tax authority could pick out 
individuals for audit and guess the particulars of their tax return data, then 
tailor the audit decision policy accordingly. Further, if a policy must be 
approved in advance by some oversight or certification body, the policy would 
need to be decided on and implemented in software far enough in advance to 
admit certification or review. Finally, if such certification does take place, 
subjects of the policy’s decisions (or overseers of those decisions) can be 
certain after the fact that the policy which was certified is the policy which 
was actually used in practice.123 

To the extent that the invariants of interest in a computer system are simple 
enough to compute, their truth can be attested by the same zero-knowledge 
proofs that attest to the relationship between the code, the inputs, and the 
outputs. Because powerful, modern, zero-knowledge techniques can be applied 
to any code, they can also be applied to code that performs the analysis of these 


123 Electronic voting systems have suffered from such problems in practice. In many 
jurisdictions, voting system software must be certified before it can be used in polling places. 
Systems are tested by the Election Assistance Commission (EAC), an independent commission 
created by the 2002 Help America Vote Act. See Testing and Certification Program, U.S. ELECTION 
ASSISTANCE COMMISSION, http://www.eac.gov/testing and certification [https://perma.cc/8DFX- 
LTYD] (detailing the EAC’s testing and certification regime). However, in many cases, updated, 
uncertified software has been used in place of certified versions because of pressure to include 
updated functionality or bug fixes. See, e.g., Fitrakis v. Husted, No. 2:12-cv-1015, 2012 WL 5411381 
(S.D. Ohio Nov. 6, 2012) (involving a suit arising out of updates to voting systems immediately 
prior to the 2012 presidential election in Ohio). 
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invariants, and the execution of that code can be considered as part of the 
operation of the system.124 

Simply publishing commitments to the inputs and outputs of a system 
rather than making them transparent will not solve all of the issues with 
transparency brought up in Part I. However, it can address the need for 
legitimate secrecy of the system, its inputs, or its outputs. Because it is 
possible, using these methods, to verify that a particular input and a particular 
decision policy correspond to a particular output, it is not strictly necessary 
to see these values in order to investigate the system’s procedural regularity. 

We describe how certification of procedural regularity can be done for 
randomized software, such as software implementing a lottery, in greater 
detail in Section II.E below. Later, in Part III, we explain how these tools can 
extend to certify other, more complicated invariant properties of interest, 
enabling proof that a system comports with substantive goals or principles 
beyond simple procedural regularity. 


E. Applying Technical Tools to Reform the Diversity Visa Lottery 


Armed with these tools, we can turn to the question of how to ensure the 
procedural regularity of automated decisionmaking. To illustrate how 
designing a computer system can make it more accountable, we will apply the 
methods described above to a case study: the Diversity Visa Lottery (DVL) 
operated by the U.S. Department of State. 


1. Current DVL Procedure 


The DVL is run annually by the State Department to grant U.S. 
permanent resident visas (“green cards”) to 50,000 immigrants from around 
the world. The process, prescribed by 8 U.S.C. § 1153(c), is intended to increase 
the national and regional diversity of immigrants to the United States by 
granting visas to a sample of people from countries otherwise underrepresented 
in the immigrant population. 

The annual DVL process operates as follows.125 Would-be immigrants 
apply to be entered in the lottery, applicants are grouped according to their 
country of birth, and countries are assigned to one of six regional groups. 
Within each group, applicants are put into a rank-ordered list in a random 
order (the lottery step). The Attorney General then calculates the number of 


124 For example, suppose that we wish to demonstrate that a decision would be the same if the 
subject’s gender were reversed. The software implementing the decision could simply compute the 
decision with a different gender for each subject and confirm that the same result is reached in each case. 

125 See generally Immigration and Nationality Act § 203(c), 8 U.S.C. § 1153(c) (2012); U.S. 
DEP’T OF STATE, FOREIGN AFFAIRS MANUAL ch. 9, § 502.6. 
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applicants to accept from each region using a formula based on the number 
of immigrants to the United States in recent years from each other region. 
The calculated number of applicants is selected from the top of each group’s 
rank-ordered list. These “winners” are screened for eligibility to enter the 
United States, and they receive visas if they are eligible. In some years, 
additional winners are selected so that all statutorily available visas can 
eventually be awarded, even if some applicants fail the screening process, 
drop out, or fail to proceed with their visa application.126 

Questions have been raised about the correctness and accountability of 
this process. Would-be immigrants sometimes question whether the process 
is truly random or, as some suspect, is manipulated in favor of individuals or 
groups favored by the U.S. government. This suspicion, in turn, may subject 
DVL winners to reprisals, on the theory that winning the DVL is evidence of 
having collaborated secretly with U.S. agencies or interests. 

There have also been undeniable failures in carrying out the DVL process. 
For example, the 2012 DVL initially reported incorrect results due to 
programming errors coupled with lax management.17 An accountable 
implementation of the DVL could address both issues by demonstrating that 
there is no favoritism in the process and by making it easy for outsiders to 
check that the process was executed correctly. 


2. Transparency Is Not Enough 


The DVL is an automated decision system for which transparency alone 
cannot solve its problems. First, the software implementing the decisions 
appears to be written in an irreproducible way.128 The system relies on the 
computer’s operating system to provide random numbers; thus, attempts to 
replicate the program’s execution at another time or on another computer will 
yield different random numbers and therefore a different DVL result. 
Notably, no amount of reading, analyzing, or testing of the software can 
remedy the nonreplicability of this software. 

Second, the privacy interests of participants bar full transparency. People 
who apply to the DVL do not want their information, or even the fact that 
they applied, to be published. However, such publication is needed for the 
process to be verified through transparency and auditing. The State 


126 Visa Bulletin for June 2015, U.S. DEPT ST. BUREAU CONSULATE AFF., https://travel.state. 
gov/content/visas/en/law-and-policy/bulletin/2015/visa-bulletin-for-june-2015.html [https://perma.cc/ 
7H7L-SJKX]. 

127 Memorandum from Howard W. Geisel, Deputy Inspector Gen., U.S. Dept of State, 
Review of the FY 2012 Diversity Visa Program Selection Process, ISP-I-12-01 (Oct. 13, 2011), https:// 
oig.state.gov/system/files/176330.pdf [https://perma.cc/4FWM-URY]]. 

128 Id. 
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Department could try to work around this problem by assigning an opaque 
record ID to each applicant and then having the lottery choose record IDs 
rather than applicants, but lottery operators could manipulate the outcome 
by retroactively assigning winning record IDs to people they wanted to favor. 
Further, it would be difficult to verify that no extra record IDs corresponding 
to actual participants had been added. 


3. Designing the DVL for Accountability 


Instead of this inherently unverifiable approach, we propose a technical 
solution for building an accountable version of the DVL.129 Using the 
techniques we have described, the State Department can demonstrate that it 
is running a fair lottery among a hidden set of participants. 130 

We can solve the nonreplicability problem by choosing a random seed, as 
described previously. The third-party generating the random value used to 
create the seed could be one or more trusted NGOs, or applicants could 
provide a “PIN” on their applications. 

Recall that the decision policy for the DVL is fixed in statute and hence 
publicly known. To provide oversight, the State Department could publish in 
the Federal Register a commitment to its software source code (far in advance 
of any decisions being made) and a commitment to all the inputs (i.e., to each 
data element in an application for the US visa) used to create the rank-ordered 
list and calculate the cut-off points. The State Department would also need to 
provide a zero-knowledge proof showing that applying the committed-to 
software to the committed-to inputs produces the announced lottery results. 
The proof should also demonstrate that the commitment to the software 
published in advance of all decisions is a commitment to the same software 
as the one used in each individual decision. These actions would bind the 
State Department to its choices of software, source code, and applicant data; 
ensure that the commitment to the software was not a fake; and prove that 
the same procedure was used to render each decision. Subsequent auditing 
by an oversight body should establish that the source code in the commitment 
faithfully implements the policy specified by statute (the code should be 
designed to enable this). 


129 A full technical analysis is beyond the scope of this Article. 

130 Note that it is less straightforward to prove that the set of participants actually considered 
in the lottery matches the set of individuals who applied to be included. For example, the operator 
of the lottery might insert “shills,” or lottery entries that do not correspond to any real applicant, 
and if one of these applications were to be chosen, that place could be given improperly to anyone 
of the Department’s choosing. It is technically nontrivial to prove that no extra applications were 
considered; studies of end-to-end secure voting protocols provide methods to do so. See, e.g., Daniel 
Sandler et al., VoteBox: A Tamper-Evident, Verifiable Electronic Voting System, 2008 PROC. 17TH CONF. 
ON SECURITY SYMP. 349 (enunciating the measures necessary to make electronic voting secure). 
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Finally, the State Department should determine an adequate method for 
generating a random seed to be used in the lottery step. This method should 
guarantee to the public that it is not possible for the State Department to 
choose winners by rearranging applications.131 This could be accomplished by 
combining random data chosen at a public ceremony (as is done for state 
lotteries); alternatively, the State Department could cooperate with interested 
NGOs to produce a verifiable random seed with a random value selected 
exclusively by the State Department (and published prior to the ceremony and 
any lottery applications) along with something that identifies a particular 
lottery entry uniquely (e.g., the applicant’s full application data, reduced by 
cryptographic hash, to a small numeric value). Depending on the implementation 
and application, the State Department could also include randomness 
selected by DVL applicants on their application, which could be harvested 
passively by tracking mouse movements during the application process.132 

Once these steps are taken, each applicant can be assured that the State 
Department’s decision on his application is fully explainable. If the applicant 
has questions regarding the process or a governmental overseer wants to audit 
it, the decisions will be replicable, and, if necessary, the secret source code 
and secret input data (including the random choices made in the lottery step) 
can be revealed and verified—by a court or auditing agency—to be the proper 
code and data used to render the decision.133 

These solutions depend on both redesigning the software code (a 
technical solution) and adopting procedures relating to how the software 
program is used (a legal or policy solution). They must be deployed during 
the design of the decision process and cannot salvage a poorly designed 
system after the fact. In hindsight, it should not be surprising that the path 
to accountability for computational processes requires some redesign of the 
processes themselves. The same is true for noncomputational administrative 
processes, where the most accountable processes are those that are designed 
with accountability in mind. 


131 Random choices in the DVL must be demonstrably random even to nonparticipants so 
that winners can plausibly claim that they were chosen by lottery and not because of sympathy 
for U.S. interests. 

132 See supra note 117 and accompanying text. 

133 In fact, just as the applicant can be convinced that his decision is explainable without seeing 
the secret algorithm or secret inputs, an oversight body can be convinced that particular decisions 
were made correctly without seeing the applicant’s inputs, which might contain sensitive data like 
health records or tax returns. Thus, subsequent auditing is rendered more useful and more acceptable 
to participants, as it can determine the basis for every decision without revealing sensitive information. 
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III. DESIGNING ALGORITHMS TO ASSURE FIDELITY TO 
SUBSTANTIVE POLICY CHOICES 


In Part I, we described methods that permit certification of properties of 
computer systems, and in Part II, we demonstrated how those methods can 
ensure that automated decisions are reached in accordance with agreed upon 
rules, a goal we called procedural regularity. In this Part, we examine how 
those methods could be used to certify other system properties that 
policymakers desire. Accountability demands not only that we certify that a 
policy was applied evenly across all subjects, but also that those subjects can 
be certain that the policy furthers other substantive goals or principles. A 
subject may want to know: Is the rule correctly implemented? Is it moral, 
legal, and ethical? Does it operate in the aggregate with fidelity to substantive 
policy choices? 

We focus here on the goal of nondiscrimination,"4 in part because specific, 
additional technical tools have developed to assist with it, and in part because 
the use of automated decisionmaking already has raised concerns about 
discrimination and the ability of current legal frameworks to deal with 
technological change.135 The well-established potential for unfairness in 
systems that use machine learning, in which the decision rule itself is not 
programmed by a human but rather inferred from data, has heightened these 
discrimination concerns. However, what makes a rule unacceptably 
discriminatory against some person or group is a fundamental and contested 
question. We do not address that question here, much less claim to resolve it 
with computational precision. Instead, we describe how an emerging body of 
computer science techniques may be used to avoid outcomes that could be 
considered discriminatory. 

Fidelity to policy choices like nondiscrimination is a more complicated 
goal than procedural regularity, and the solutions that currently exist to 
address it are less robust. Technical tools offer ways to ameliorate these 
problems, but they generally require a well-defined notion of what sort of 
fairness they are supposed to be enforcing. In this Part, we outline a few 
proposed well-defined notions. We present these techniques as examples of 
system properties that could be certified using the techniques described in 


134 The word “discrimination” carries a very different meaning in engineering conversations 
than it does in public policy. Among computer scientists, the word is a value-neutral synonym for 
differentiation or classification: a computer scientist might ask, for example, how well a facial 
recognition algorithm successfully discriminates between human faces and inanimate objects. But, 
for policymakers, “discrimination” is most often a term of art for invidious, unacceptable distinctions 
among people—distinctions that either are, or reasonably might be, morally or legally prohibited. 
We use the latter meaning here. 

135 See PASQUALE, supra note 46, at 8-9 (describing the problem of discrimination through the 
use of automated decisionmaking). 
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Part I, but we do not necessarily advocate for any of them; ultimately, 
policymakers must decide whether these properties or others square with 
nondiscrimination goals. 

In addition, the precision of computer code often brings into sharp focus 
the tensions within current legal frameworks for antidiscrimination. Computers 
favor hard and fast rules over the types of standards and balancing tests often 
found in our common law system and civil rights law. While these 
characteristics of the current legal approach suggest that doctrinal reform may 
be necessary to apply computerized decisionmaking in an area, we are not 
advocating a policy regime entirely made of bright-line rules or predetermined 
fairness criteria. In fact, we believe that investigations of fairness should always 
be in the purview of ex post review processes. Instead, we offer an overview 
of the problem of algorithmic discrimination, the current state of the related 
technical tools, and the relationship of these tools to existing legal 
frameworks. We describe the types of properties that can be specified in 
advance and certified to be in force, even if none of the properties is sufficient 
on its own to guarantee compliance with a policy regime. Our aim is to both 
elucidate the current state of the art and suggest directions for further 
research and action. 


A. Machine Learning, Policy Choices, and Discriminatory Effects 


We focus here on decisions developed through machine learning—on 
situations where a machine has been “trained” through exposure to a large 
quantity of data and infers a rule from the patterns it observes. Computers 
are especially well-suited to discover patterns in these input-output pairs that 
can then guide future decisionmaking. In contrast to human-made rules, these 
rules for decisionmaking are induced from historical examples—they are, 
quite literally, rules learned by example. Humans orchestrate a computerized 
rule-creation process, rather than imparting the rules directly. These kinds of 
decisions raise problems for the methods described in Part I because the 
system's designer does not fix the decision rule directly, and, as a result, the 
rule cannot directly be verified in the manner we have described. Instead, for 
the tools to show that such systems meet policy goals, policymakers must 
determine the substantive properties that the systems should have, and, if 
such properties exist, the tools of Parts I and II can then be used to 
demonstrate that techniques for certifying such properties are in use and 
implementers can then design the systems to allow the certification of these 
properties and permit the type of accountability we have proposed above. 

Machine learning is an increasingly common approach to solving problems 
that once seemed computationally intractable due to their complexity (e.g., 
object recognition in a photograph). The recent movement of software systems 
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into a growing number of domains owes primarily to successful applications 
of machine learning, which is thus the primary focus of our analysis. 

A significant concern about automated decisionmaking is that it may 
simultaneously systematize and conceal discrimination. Because it can be 
difficult to predict the effects of a rule in advance (especially for large, 
complicated rules or rules that are machine-derived from data), regulators 
and observers may be unable to tell that a rule has discriminatory effects. In 
addition, decisions made by computers may enjoy an undeserved assumption 
of fairness or objectivity.36 However, the design and implementation of 
automated decision systems can be vulnerable to a variety of problems that 
can result in systematically faulty and biased determinations.137 

These decision rules are machine-made and follow mathematically from 
input data, but the lessons they embody may be biased or unfair nevertheless. 
Below, we describe a few illustrative ways that models, or decision rules 
derived from data, generated through machine learning, may turn out to be 
discriminatory. We adapt a taxonomy laid out in previous work by Solon 
Barocas and Andrew D. Selbst!38 and make use of the “catalog of discriminatory 
evils” of machine learning systems laid out by Hardtt39 and Dwork et al.140 

First, algorithms that include some type of machine learning can lead to 
discriminatory results if the algorithms are trained on historical examples that 
reflect past prejudice or implicit bias, or on data that offer a statistically 
distorted picture of groups comprising the overall population. Tainted 
training data would be a problem, for example, if a program to select among 
job applicants is trained on the previous hiring decisions made by humans, 
and those previous decisions were themselves biased.141 Statistical distortion, 


136 See Paul Schwartz, Data Processing and Government Administration: The Failure of the 
American Legal Response to the Computer, 43 HASTINGS L.J]. 1321, 1342 (1992) (describing the deference 
that individuals give to computer results as the “seductive precision of output”). 

137 See id. at 1342-43 (noting that the computer creates “new ways to conceal ignorance and 
subjectivity” because people overestimate its “accuracy and applicability”). 

138 See Barocas & Selbst, supra note 8 (describing a taxonomy that isolates specific technical 
issues to create a decisionmaking model that may disparately impact protected classes). 

139 Moritz A.W. Hardt, A Study of Privacy and Fairness in Sensitive Data Analysis (Nov. 2011) 
(unpublished Ph.D. dissertation, Princeton University) (on file with author). 

140 Cynthia Dwork et al., Fairness Through Awareness, 2012 PROC. 3RD INNOVATIONS 
‘THEORETICAL COMPUTER SCI. CONF. 214. 

141 See Barocas & Selbst, supra note 8, at 682 (citing Stella Lowry & Gordon Macpherson, 4 
Blot on the Profession, 296 BRIT. MED. J. 657, 657 (1988)) (describing how a hospital developed a 
computer program to sort medical school students based on previous decisions that had disfavored 
racial minorities and women). Another example is a Google algorithm that showed ads for arrest 
records much more frequently when black-identifying names were searched than when white- 
identifying names were searched—likely because users clicked more often on arrest record ads for 
black-identifying names and the algorithm learns from this behavior with the purpose of maximizing 
click-throughs. Id. at 682-83 (citing Latanya Sweeney, Discrimination in Online Ad Delivery, COMM. 
ACM, May 2013, at 44, 47 (2013)). 


2017 | Accountable Algorithms 681 


even if free of malice, can produce similarly troubling effects: consider, for 
example, an algorithm that instructs police to stop and frisk pedestrians. If 
this algorithm has been trained on a dataset that overrepresents the incidence 
of crime among certain groups (because these groups have historically been 
the target of disproportionate enforcement), the algorithm may direct police 
to detain members of these groups at a disproportionately high rate (and 
nonmembers at a disproportionately low rate). Such was the case with the 
New York City Police Department’s stop-and-frisk program, for which data 
from 2004 to 2012 showed that 83% of the stops were of black or Hispanic 
persons and 10% were of white persons in a resident population that was 52% 
black or Hispanic and 33% white.142 Note that the overrepresentation of black 
and Hispanic people in this sample may lead an algorithm to associate 
typically black or Hispanic traits with stops that lead to crime prevention, 
simply because those characteristics are overrepresented in the population 
that was stopped.143 

Second, machine learning models can build in discrimination through 
choices in how models are constructed. Of particular concern are choices 
about which data models should consider, a problem computer scientists call 
feature selection. Three types of choices about inputs could be of concern: using 
membership in a protected class directly as an input (e.g., decisions that take 
gender into account explicitly); considering an insufficiently rich set of 
factors to assess members of a protected class with the same degree of 
accuracy as nonmembers (e.g., in a hiring application, if fewer women have 
been hired previously, data about female employees might be less reliable 
than data about male employees); and relying on factors that happen to serve 
as proxies for class membership (e.g., women who leave a job to have children 
lower the average job tenure for all women, causing this metric to be a known 
proxy for gender in hiring applications). Eliminating proxies can be difficult, 
because proxy variables often contain other useful information that an analyst 
wishes the model to consider (for example, zip codes may indicate both race 
and differentials in local policy that is of legitimate interest to a lender). The 
case against using a proxy is clearer when alternative inputs could yield 
equally effective results with fewer disadvantages to protected class members. 
A problem of insufficiently rich data might be remedied in some cases by 
gathering more data or more features, but if discrimination is already 
systemic, new data will retain the discriminatory impact. While it is tempting 


142 David Rudovsky & Lawrence Rosenthal, Debate: The Constitutionality of Stop-and-Frisk in 
New York City, 162 U. PA. L. REV. ONLINE 117, 120-21 (2013). 

143 The underrepresentation of white people would likely cause the opposite effect, though it 
could be counter-balanced if, say, the police stopped a subset of white people who were significantly 
more likely to be engaged in criminal behavior. 
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to say that technical tools could allow perfect enforcement of a rule barring 
the use of protected attributes, this may in fact be an undesirable policy 
regime. As previously noted, there may be cases where allowing an algorithm 
to consider protected class status can actually make outcomes fairer. This may 
require a doctrinal shift, as, in many cases, consideration of protected status 
in a decision is presumptively a legal harm. 

Third and finally, there is the problem of “masking”: intentional 
discrimination disguised as one of the above-mentioned forms of unintentional 
discrimination. A prejudiced decisionmaker could skew the training data or 
pick proxies for protected classes with the intent of generating discriminatory 
results.144 More pernicious masking could occur at the level of designing a 
machine learning model, which is a very human-driven, exploratory process.145 


B. Technical Tools for Nondiscrimination 


As mentioned in the previous Part, transparency and after-the-fact 
auditing can only go so far in preventing undesired results. Ideally, those 
types of ex post analyses should be used in tandem with powerful ex ante 
techniques during the design of the algorithm. The general strategy we 
proposed in Section II. D—publishing commitments and using zero-knowledge 
proofs to ensure that commitments correspond to the system's decisionmaking 
actions—can certify any property of the decision algorithm that can be checked 
by a second examination algorithm.14 Such properties can be verified by 
making the examination algorithm public and giving a zero-knowledge proof 
that, if the examination algorithm were run on the secret decision algorithm, 
it would report that the decision algorithm has the desired property. The 
question then is which, if any, properties policymakers would want to build 
into particular decision systems. 

A simple example of such a property would be the exclusion of a certain 
input from the decisionmaking process. A decisionmaker could show that a 
particular algorithm does not directly use sensitive or prohibited classes of 
information, such as gender, race, religion, or medical status. 

The use of machine learning adds another wrinkle because decision rules 
evolve on the fly—they are not specified directly, but are inferred from the 
data. However, the absence of static, predetermined decision rules does not 
necessarily preclude the use of our certification strategy. Computer scientists, 


144 See Barocas & Selbst, supra note 8, at 692-93 (describing ways to intentionally bias data 
collection in order to generate a preferred result). 

145 In other words, the machine learning model would be intentionally coded to develop bias. 

146 Such an algorithm might be a tool for verifying properties of software or simply a software 
test. See supra Part I (discussing software testing and software verification in greater detail). 
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including Hardt,47 Dwork et al.,148 and others, have developed techniques 
that formalize fairness in such a way that they can constrain the machine 
learning process so that learned decision rules have specific well-defined 
fairness properties. These methods also can be incorporated in the design of 
systems such that their inclusion in the decisionmaking process implies the 
incorporation of fairness properties that can be certified and proven.¥9 

We describe three such methods below. First, decision systems can 
incorporate randomness to maximize the gain of learning from experience. 
Second, computer science offers many emerging approaches to maximize 
fairness, defined in a variety of ways, in machine learning systems. At a high 
level, all of these definitions reduce to the proposition that similarly situated 
people should be treated similarly, without regard to sensitive attributes. As 
we shall see, simple blindness to these attributes is not sufficient to guarantee 
even this simplified notion of fairness. Finally, related ideas from differential 
privacy can also be used to guarantee that protected status could not have 
been a substantial factor in certain decisions. 

Those who use algorithmic decisionmaking today regularly make assertions 
about properties of these algorithms without proving them. This likely occurs 
because they are required by law to disclose certain facts about their decision 
process to regulators and consumers,!5° they simply want to generate good 
will, or they demonstrate better behavior than a competitor. But without 
proof, these assertions are just words on paper, subject to challenge by 
skeptical regulators and disbelief by skeptical consumers. This skepticism is 
not entirely unfounded: these assertions have proved false in the past. Digital 
evidence, such as zero-knowledge proofs, gives a direct connection between 
the fact being asserted and the technical mechanism of decisionmaking. This 
proof provides the consumer with a high assurance that the assertion 
proffered relates meaningfully to the facts on the ground. 


1. Learning from Experience 


As mentioned in Section I.B, incorporating randomness into an algorithm 
can give it flexibility to operate outside of the environment for which it was 
designed. Similarly, randomness can prevent hidden biases in the design or 


147 Hardt, supra note 139. 

148 Dwork et al., supra note 140. 

149 We concentrate on certification and proof of a system property to an overseer, observer, or 
participant. However, these tools are also valuable for compliance (since proofs can certify to the 
implementer of a system that the system is working as intended) and for demonstration that a 
decisionmaker will be able to show how and why he or she used certain data after the fact in case of 
an audit or dispute. 

150 See, eg., 12 C.F.R §§ 203.4-5 (2015) (providing requirements for the compilation, 
disclosure, and reporting of loan data). 
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deployment of an algorithm from leading to consistent discriminatory 
outcomes. There is a large and rich literature on how to maximally learn from 
previous data and how to use random choices to ensure that a model is as 
faithful as possible to the real world.151 

Consider a machine learning algorithm for hiring that is trained using a 
biased set of initial data indicating that women are weak candidates, even 
though gender does not predict job performance among the full population. 
If the resulting model would hire mostly men, the algorithm for hiring can 
create a self-fulfilling prophecy in which it finds that characteristics of 
successful hires correlate strongly with proxies for gender. But, if the 
algorithm is designed to incorporate an element of randomness such that 
some candidates who are not predicted to do well get hired (and have their 
performance tracked), the validity of the initial assumptions can be tested and 
the accuracy and fairness of the entire system will benefit over time. By 
occasionally guessing about candidates for whom the model cannot make 
confident predictions, the model can gather additional data and evolve to 
become more faithful to the real world. 

Similarly, randomness is often necessary when training machine learning 
models. Models may become too specialized or specific to the data used for 
training, a problem called “overfitting.” Making random choices during the 
model's training process can prevent this problem. Likewise, models may find 
a decision rule is well-suited for some portion of the input, but not the best 
rule overall. Randomness can also help avoid this bias. Consider, for example, 
a credit-scoring model trained initially on a biased set of data that underrates 
the creditworthiness of a minority group. Even if the model is the best 
possible decision rule for a majority of the population similar to the biased 
input data, the model may unfairly deny access to credit to members of that 
minority group. In addition to the discrimination, the use of this model would 
deny creditors business opportunities with the unfairly rejected individuals. 
Here again, designing the model to occasionally guess randomly, while 
tracking expected versus actual performance, could improve the model’s 
faithfulness to the population on which it is actually used rather than the 
biased population on which it was trained. The information from this 
injection of randomness can be fed back to the model to improve the accuracy 
and fairness of the system overall. 


151 This literature is divided between the machine learning research community in computer 
science and the study of optimal decisionmaking in statistics. See supra note 63. 
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2. Fair Machine Learning 


One commonly understood way to demonstrate that a decision process is 
independent of sensitive attributes is to preclude the use of those sensitive 
attributes from consideration. For example, race, gender, and income may be 
excluded from a decisionmaking process to assert that the process is “race-blind,” 
“gender-blind,” or “income-blind.”152 From a technical perspective, however, this 
approach is naive. Blindness to a sensitive attribute has long been recognized as 
an insufficient approach to making a process fair. The excluded or “protected” 
attributes can often be implicit in other nonexcluded attributes. For example, 
even when race is excluded as a valid criterion for a credit decision, redlining 
may occur when a zip code is used as proxy that closely aligns with race.153 

This type of input “blindness” is insufficient to assure fairness and 
compliance with substantive policy choices. Although there are many 
conceptions of what defines fairness, we consider here a definition of fairness 
in which similarly situated people are given similar treatment—that is, a fair 
process will give similar participants a similar probability of receiving each 
possible outcome. This is the core principle of a developing literature on fair 
classification in machine learning, an area first formalized by Dwork, Hardt, 
Pitassi, Reingold, and Zemel.54 This work stems from a longer line of 
research on mechanisms for data privacy.‘ We further describe the 
relationship between fairness in the use of data and privacy below. 

The principle that similar people should be treated similarly is often 
called individual fairness, and it is distinct from group fairness in the sense that 
a process can be fair for individuals without being fair for groups.156 Although 
it is almost certainly more policy-salient, group fairness is more difficult to 
define and achieve. The most commonly studied notion of group fairness is 
statistical parity, the idea that an equal fraction of each group should receive 


152 See, e.g., 12 C.F.R. § 1002.5(b) (2015) (“A creditor shall not inquire about the race, color, 
religion, national origin, or sex of an applicant or any other person in connection with a credit 
transaction.”); id. § 1002.6(b)(g) (“[A] creditor shall not consider race, color, religion, national 
origin, or sex (or an applicant’s or other person’s decision not to provide the information) in any 
aspect of a credit transaction.”). 

153 See Jessica Silver-Greenberg, New York Accuses Evans Bank of Redlining, N.Y. ‘TIMES: 
DEALBOOK (Sept. 2, 2014), http://dealbook.nytimes.com/2014/09/02/new-york-set-to-accuse-evans- 
bank-of-redlining [https://perma.cc/3YFA-6Na4J] (detailing a redlining accusation in great detail). 

154 Dwork et al., supra note 140. 

155 Specifically, the work of Dwork et al. is a generalization of ideas originally presented in 
Cynthia Dwork, Differential Privacy, 2006 PROC. 33RD INT'L COLLOQUIUM ON AUTOMATA, 
LANGUAGES & PROGRAMMING 1. As discussed below, fairness can be viewed as the property that 
sensitive or protected status attributes cannot be inferred from decision outcomes, which is very 
much a privacy property. 

156 Sometimes, a more restrictive notion of individual fairness implies group fairness, Id. 
Intuitively, this is because if people who are sufficiently similar are treated sufficiently similarly, 
there is no way to construct a minority of people who are treated in a systematically different way. 
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each possible outcome. While statistical parity seems like a desirable policy 
because it eliminates redundant or proxy encodings of sensitive attributes, it 
is an imperfect notion of fairness. For example, statistical parity says nothing 
about whether a process addresses the “right” subset of a group. Imagine an 
advertisement for an expensive resort: we would not expect that showing the 
advertisement to the same number of people in each income bracket would 
lead to the same number of people clicking on the ad or buying the associated 
product. For example, a malicious advertiser wishing to exclude a minority 
group from a resort could design its advertising program to maximize the 
likelihood of conversion for the desired group while minimizing the likelihood 
that the ad will result in a sale to the disfavored group. In the same vein, if a 
company aimed to improve the diversity of its staff by offering the same 
proportion of interviews to candidates with minority backgrounds as are 
minority candidate applications, that is no guarantee that the number of 
people hired will reflect the population of applicants or the population in 
general. And the company could hide discriminatory practices by inviting 
only unqualified members of the minority group to apply, effectively creating 
a self-fulfilling prophecy for decision rules established by machine learning. 
The work of Dwork et al. identifies an additional interesting problem with 
the “fairness through blindness” approach: by remaining blind to sensitive 
attributes, a classification rule can select exactly the opposite of what is 
intended.157 Consider, for example, a system that classifies profiles in a social 
network as representing either real or fake people based on the uniqueness of 
their names. In European cultures, from which a majority of the profiles 
come, names are built by making choices from a relatively small set of possible 
first and last names, so a name which is unique across this population might 
be suspected to be fake. However, other cultures (especially Native American 
cultures) value unique names, so it is common for people in these cultures to 
have names that are not shared with anyone else. Since a majority of accounts 
will come from the majority of the population, for which unique names are 
rare, any classification based on the uniqueness of names will inherently 
classify real minority profiles as fake at a higher rate than majority profiles,158 
and may also misidentify fake profiles using names drawn from the minority 
population as real. This unfairness could be remedied if the system were 
“aware” of the minority status of a name under consideration, since then the 


157 See Dwork et al., supra note 140. 
158 That is, the minority group will have a higher false positive rate. 
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algorithm could know whether the implication of a unique name is that a 
profile is very likely to be fake or very likely to be real.159 

This insight explains why the approach taken by Dwork et al. is to enforce 
similar probabilities of each possible outcome on similar people, requiring 
that the aggregate difference in probability of any individual receiving any 
particular outcome be limited.160 Specifically, Dwork et al. require that this 
difference in chance of outcome be less than the difference between 
individuals subject to classification.16t This requires a mathematically precise 
notion of how “different” people are, which might be a score of some kind or 
might naturally arise from the data in question.162 This notion of similarity 
must also capture all relevant features, including possibly sensitive or protective 
attributes such as minority status, gender, or medical history. Because this 
approach requires the collection and explicit use of sensitive attributes, the 
work describes its definition of fairness as fairness through awareness.163 While 
the work of Dwork et al. provides only a theoretical framework for building 
fair classifiers, others have used it to build practical systems that perform 
almost as well as classifiers that are not modified for fairness. 

The work of Dwork et al. also provides the theoretical basis for a notion 
of fair affirmative action, the idea that imposing an external constraint on the 
number of people from particular subgroups who are given particular 
classifications should have a minimal impact on the principle that similar 
people are treated similarly. This provides a technique for forcing a fairness 
requirement such as statistical parity even when it will not arise naturally 
from some classifier. 

A more direct approach to making a machine learning process fair is to 
modify or select the input data in such a way that the output satisfies some 
fairness property. For example, in order to make sure that a classifier does not 
over-reflect the minority status of some group, we could select extra training 
samples from that group or duplicate samples we already have. In either case, 


159 In this case, differential treatment based on a protected status attribute improves the 
performance of the automated decision system in a way that requires that the system know and make 
use of the value of that attribute. 

160 See Dwork et al., supra note 140, at 215 (explaining that fairness can be captured under the 
principle that “two individuals who are similar with respect to a particular task should be classified 
similarly”). 

161 This is formalized as the proposition that the difference in probability distributions between 
outcomes for each subgroup of the population being classified is less than the difference between 
those groups, for a suitable measurement of the difference between groups. For technical reasons, 
this particular formulation is mathematically convenient, although different bounds might also be 
useful. For the formal mathematical definition, see id. at 216. 

162 For example, if the physical location of subjects is a factor in classification, we might 
naturally use the distance between subjects as one measure of their similarity. 

163 Dwork et al., supra note 140, at 215. 
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care must be taken to avoid biasing the training process in some other way or 
overfitting the model to the nonrepresentative data. 

Other work focuses on fair representations of data sets. For example, we can 
take data points and assign them to clusters, or groups of close-together points, 
treating each cluster as a prototypical example of some portion of the original 
data set. This is the approach taken by Zemel, Wu, Swersky, Pitassi, and 
Dwork.164 Specifically, Zemel et al. show how to generate such prototypical 
representations automatically and in a way that guarantees statistical parity 
for any subgroup in the original data. In particular, the probability that any 
person in the protected group is mapped to any particular prototype is equal 
to the probability that any person not from the protected group is mapped to 
the same prototype.165 Therefore, classification procedures which have access 
only to the prototypes must necessarily not discriminate, since they cannot 
tell whether the prototype primarily represents protected or unprotected 
individuals. Zemel et al. test their model on many realistic data sets, including 
the Heritage Health Prize data set, and determine that it performs nearly as 
well as best-of-breed competing methods while ensuring substantial levels of 
fairness.166 This technique allows for a kind of “fair data disclosure,” in which 
disclosing only the prototypes allows any sort of analysis, fair or unfair, to be 
run on the data set to generate fair results. 

A related approach is to use a technique from machine learning called 
regularization, which involves modifying the model training process to yield 
models that are more generalizable by introducing a penalty associated with 
undesirable model attributes or behaviors. This approach has also led to many 
useful modifications to standard tools in the machine learning repertoire, 
yielding effective and efficient fair classifiers.167 

The work of Zemel et al. suggests a related approach, which is also used 
in practice: the approach of generating fair synthetic data. Given any set of 
data, we can generate new data such that no classifier can tell whether a 
randomly chosen input was drawn from the real data or the fake data. 
Furthermore, we can use approaches like that of Zemel et al. to ensure that 
the new data are at once representative of the original data and also fair for 
individuals or subgroups. Because synthetic data are randomly generated, 
they are useful in situations where training a classifier on real data would 
create privacy concerns. Also, synthetic data can be made public for others to 


164 Richard Zemel et al., Learning Fair Representations, 28 PROC. 30TH INT'L CONF. ON 
MACHINE LEARNING 325 (2013). 

165 Id. 

166 Id. 

167 See, eg., Toshihiro Kamishima et al., Fairness-Aware Learning Through Regularization 
Approach, 2011 PROC. 3RD IEEE INT'L WORKSHOP ON PRIVACY ASPECTS DATA MINING 643 
(describing a model in which two types of regularizers were adopted to enforce fair classification). 
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use, although care must be taken to avoid allowing others to infer facts about 
the underlying real data. Such model inversion attacks68 have been 
demonstrated in practice, along with other inference or deanonymization attacks 
that allow sophisticated conclusions without direct access to the actual data 
that give rise to the conclusions.169 

All of these approaches demonstrate that it is possible to build a wide 
range of definitions of fairness into a wide variety of data analysis and 
classification systems, at least to the extent that a definition of fairness is 
known or can be approximated in advance. There are no bright-line rules that 
allow the designer or operator of a machine learning system to guarantee that 
the system’s behavior is compliant with antidiscrimination laws. Nor do we 
believe that such rules can or even should exist. It is not for technologists to 
define an ex ante notion of fairness that applies in all cases or even just for a 
specific system. Rather, fairness must be determined contextually and often 
must be reviewed ex post. Regardless, it is certainly not impossible to build 
fairness into automated decision systems, which shows that unconstrained use 
of data analysis is not always necessary. Uses of data that do not employ 
methods to investigate or ensure fairness must account for their decision 
policies in some other way. 

Many of these approaches rely on the insufficient notion of group fairness 
by statistical parity. To the extent that more technical research can help to 
address the problem of unfairness in big data analysis, it is by expanding the 
repertoire of definitions of group fairness that can be usefully applied in 
practice and by providing better exploratory and explanatory tools for 
comparing different notions of fairness. From a public policy perspective, it 
would be extremely useful to system designers to have a set of rules, 
standards, or best practices that explain what notions of fairness should be 
used in specific real-world applications. 

A complementary notion to machine learning systems that can guarantee 
prespecified, formal fairness properties is the work of Rudin on machine 
learning systems that are interpretable. Such systems generate models that 
can be used to classify individuals, but also explanations for why those 


168 See Matthew Fredrikson et al., Privacy in Pharmacogenetics: An End-to-End Case Study of 
Personalized Warfarin Dosing, 2014 PROC. 23RD USENIX SECURITY SYMP. 17 (describing privacy 
risks in which attackers can predict a patient’s genetic markers if provided with the model and some 
demographic information). 

169 For an overview of these techniques, see Arvind Narayanan & Edward W. Felten, No Silver 
Bullet: De-identification Still Doesnt Work (July 9, 2014), http://randomwalker.info/publications/ 
no-silver-bullet-de-identification.pdf [https://perma.ce/VT2G-7ACG], and Arvind Narayanan et 
al, A Precautionary Approach to Big Data Privacy (Mar. 19, 2015), http://randomwalker.info/ 
publications/precautionary.pdf [https://perma.cc/FQR3-2MMz2]. 

170 Cynthia Rudin, Algorithms for Interpretable Machine Learning, 2014 20TH ACM SIGKDD 
CONF. ON KNOWLEDGE DISCOVERY & DATA MINING 1519. 
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classifications were made. These explanations can be reviewed later to 
understand why the model behaves a certain way, and in some cases how 
changes in the input data would affect the model’s decision. These 
explanations can be extremely valuable to experts and oversight authorities, 
who wish to avoid treating models as black boxes. 


3. Discrimination, Data Use, and Privacy 


A different way to define whether a classification is fair is to say that we 
cannot tell from the outcome whether the subject was a member of a 
protected group or not. That is, if an individual’s outcome does not allow us 
to predict that individual’s attributes any better than we could by guessing 
them with no information, we can say that outcome was assigned fairly. To 
see why this is so, observe the contrary: if the fact that an individual was 
denied a loan from a particular bank tells you that this individual is more 
likely to live in a certain neighborhood, this implies that you hold a strong 
belief that the bank denies credit to residents of this neighborhood and hence 
a strong belief that the bank makes decisions based on factors other than the 
objective credit risk presented by applicants. 

Thus, fairness can be seen as a form of an information hiding requirement 
similar to privacy. If we accept that a fair decision does not allow us to infer 
the attributes of a decision subject, we are forced to conclude that fairness is 
protecting the privacy of those attributes. 

Indeed, it is often the case that people are more concerned that their 
information is used to make some decision or classify them in some way than 
they are that the information is known or shared. This concern relates to the 
famous conception of privacy as the “right to be let alone,” in that generally 
people are concerned with the idea that disclosure interrupts their enjoyment 
of an “inviolate personality.”17! 

Data use concerns also surface in the seminal work of Solove, who refers 
to concerns about “exclusion” in “information processing,” or the lack of 
disclosure to and control by the subject of data processing and “distortion” of 
a subject’s reputation by way of “information dissemination.”!72 Solove argues 
that these problems can be countered by giving subjects knowledge of and 
control over their own data.173 In this framework, the predictive models of 
automated systems, which might use seemingly innocuous or natural 
behaviors as inputs, create anxiety on the part of data subjects. We propose a 
complementary approach: if a system’s designer can prove to an oversight 


171 Samuel D. Warren & Louis D. Brandeis, The Right to Privacy, 4 HARV. L. REV. 193, 205 (1890). 

172 Daniel J. Solove, 4 Taxonomy of Privacy, 154 U. PA. L. REV. 477, 521, 546 (2006). 

173 See id. at 546 (detailing privacy statutes that allow individuals to access and correct 
information that is maintained by government agencies). 
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entity or to each data subject that the sorts of behaviors that cause these 
anxieties are simply not possible behaviors of the system, then the use of these 
data will be more acceptable. 

We can draw an analogy between data analysis and classification problems 
and the more familiar data aggregation and querying problems that are much 
discussed in the privacy literature. Decisions about an individual represent 
(potentially private) information about that individual (i.e., one might infer 
the input data from the decision or the decision itself may be of a private 
nature), and this raises concerns for privacy. In essence, privacy may be at 
risk from an automated decision that reveals sensitive information just like 
fairness may be at risk from an automated decision. In this analogy, a vendor 
or agency using a model to draw automated decisions wants those decisions 
to be as accurate as possible, corresponding to the idea in privacy that it is 
the goal of a data analyst to build as complete and accurate a picture of the 
data subject as is feasible. 

A naive approach to making a data set private is to delete “personally 
identifying information” from the data set. This is analogous to the current 
practice of making data analysis fair by removing protected attributes from 
the input data. However, both approaches fail to provide their promised 
protections.174 The failure in fairness is perhaps less surprising than it is in 
privacy—discrimination law has known for decades about the problem of 
proxy encodings of protected attributes and their use for making inferences 
about protected status that may lead to adverse, discriminatory effects.175 

The work of Hardtt7 relates the work on fairness by Dwork et al.177 to the 
work on differential privacy by Dwork.178 As differential privacy is a well-founded 
notion of protection against inferences and the recovery of an individual identity 
from “anonymous” data, so are formal fairness properties for automated decision 
systems sound notions of fairness for individuals and a theoretical framework 
on which to ground more complicated notions of fairness for protected groups. 


174 Reidentification of individuals based on inferences from disparate data sets is a growing 
and important concern that has spawned a large literature in both computer science and law. See 
Ohm, supra note 71, at 1704 (arguing that developments in computer science demonstrate that “/d]ata 
can be either useful or perfectly anonymous but never both,” and that such developments should “trigger 
a sea change” in legal scholarship). 

175 For example, the law explicitly forbids the (sole) use of certain attributes that are likely to 
be highly correlated with protected status categories, as in protections against redlining. See, e.g., 12 
C.E.R. § 1002.5(b) (2015) (“A creditor shall not inquire about the race, color, religion, national origin, 
or sex of an applicant or any other person in connection with a credit transaction.”); id. 
§ 1002.6(b)(g) (“[A] creditor shall not consider race, color, religion, national origin, or sex (or an 
applicant’s or other person’s decision not to provide the information) in any aspect of a credit 
transaction.”). 

176 Hardt, supra note 139. 

177 Dwork et al., supra note 140. 

178 Dwork, supra note 155. 
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The many techniques of building fair data analysis and classification 
systems described above mostly require decisionmakers to have access to 
protected status information, at least during the design phase of the 
algorithm. However, in many cases, concerns about misuse, reuse, or abuse of 
this information have led to a policy regime where decisionmakers are 
explicitly barred from using such information. The deployment of these 
technical tools would require a policy change.179 The techniques described 
above could be used to make such a change less prone to engendering the very 
real concerns of data abuses that have led to the current regime. 


C. Antidiscrimination Law and Algorithmic Decisionmaking 


The goal of Part II—procedural regularity—is relatively simple from a 
legal standpoint. Procedural regularity is a core idea behind due process: the 
state cannot single out an individual for a different procedure.180 An argument 
that governance measures ensuring algorithmic procedural regularity are 
required by due process is more tenuous,!8! but an agency that implements 
such measures will not risk violating a legal requirement. 

In contrast, governance of algorithms to promote nondiscrimination runs 
into the complicated field of antidiscrimination law. Here, the movement 
toward a colorblind interpretation of equal protection has created friction 
with the precedents involving disparate impact. We argue that, given the 
current state of antidiscrimination law, designing for nondiscrimination is 
important because users of algorithms may be legally barred from revising 
processes to correct for discrimination after the fact, and technical tools offer 
solutions to help. 


1. Ricci v. DeStefano: The Tensions Between Equal Protection, 
Disparate Treatment, and Disparate Impact 


Antidiscrimination law is based upon both the constitutional guarantee of equal 
protection!82 and supplemental statutory protections. Modern interpretations 


179 One example is the privacy regime created by the Health Insurance Portability and 
Accountability Act, see supra note 78, which forbids the disclosure of certain types of covered 
information beyond those for which the data subject was previously given notice and which limits 
disclosure to covered entities subject to the same restrictions. 

180 See, e.g., Arthur S. Miller, An Affirmative Thrust to Due Process of Law?, 30 GEO. WASH. L. 
REV. 399, 403 (1962) (“Procedural due process (‘adherence to procedural regularity’), as we have 
often been told by Supreme Court justices, is the very cornerstone of individual liberties.”). 

181 See Citron, supra note 6, at 1278-1300 (arguing that current procedural protections are 
inadequate for automated decisionmaking). 

182 See U.S. CONST. amend. XIV, §1 (“No State shall ... deny to any person within its 
jurisdiction the equal protection of the laws.”). The Equal Protection Clause has also been interpreted 
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of the Equal Protection Clause generally have been divided into two camps: 
those who believe in a color-blind Constitution—protecting individualized 
assessments and eschewing any evaluations based on group status—and those 
who support antisubordination attempts to remedy inequalities between 
groups.183 The general trend has been toward colorblindness.184 

For statutory measures, we will focus on Title VII of the Civil Rights Act 
of 1964.185 Under Title VII, remedies are available for disparate treatment— 
discriminatory intent or the formal application of different rules to people of 
different groups—and disparate impact—results that differ for different 
groups.!86 Algorithmic decisionmaking blurs the definitions of disparate 
treatment and disparate impact and poses a number of open questions.187 

Is it disparate treatment when the inputs used are a proxy for membership 
in a protected class? Different rules are effectively applied to different groups 
in this case, but that difference may have no effect on the outcomes.188 If the 
people responsible for a decision know that an algorithm behaves in a way 
that has disparate impact, does that mean that they intend a discriminatory 
result?89 If an algorithm generates poor outcomes for a group of people, how 
accurate does the algorithm need to be (and how carefully does the 
decisionmaker need to test alternative algorithms) before the decisionmaker 


to apply to the federal government through the Due Process Clause of the Fifth Amendment. See, e.g., 
Kenji Yoshino, The New Equal Protection, 124 HARV. L. REV. 747, 748 n.10 (2010). 

183 See, e.g., Reva B. Siegel, From Colorblindness to Antibalkanization: An Emerging Ground of 
Decision in Race Equality Cases, 120 YALE L.J. 1278, 1281 (2011) (describing this binary as the common 
interpretation of equal protection jurisprudence), 

184 See The Supreme Court, 2008 Term—Leading Cases, 123 HARV. L. REV. 153, 289 (2009) (“The 
Court’s conception of equal protection turns largely on its swing voter, Justice Kennedy, who appears 
to support a moderate version of the colorblind Constitution.”). But see Reva B. Siegel, The Supreme 
Court, 2012 Term—Foreword: Equality Divided, 127 HARV. L. REV. 1, 6 (2013) (agreeing that “[s]hifts 
in equal protection oversight . . . are continuing to grow” but arguing that these changes are “neither 
colorblind nor evenhanded” because “the Court has encouraged majority claimants to make 
discriminatory purpose arguments about civil rights law based on inferences the Roberts Court 
would flatly deny if minority claimants were bringing discriminatory purpose challenges to the 
criminal law”). 

185 42 U.S.C. §§ 2000e-e-17 (2012), Title VII applies to employment discrimination on the 
basis of race, national origin, gender, and religion. The disparate impact framework is also used for 
discrimination in housing, employment, public entity, public accommodation, and 
telecommunications (with respect to people with disabilities). See Tex. Dept of Hous. & Cmty. 
Affairs v. Inclusive Cmtys. Project, Inc., 135 S. Ct. 2507 (2015) (holding that disparate impact claims 
are cognizable under the Fair Housing Act); Lopez v. Pac. Mar. Ass’n, 657 F.3d 762 (gth Cir. 2011) 
(deciding a disparate impact claim brought under the Americans with Disabilities Act). 

186 See Richard Primus, The Future of Disparate Impact, 108 MICH. L. REV. 1341, 1350-51 & n.56 
(2010) (describing the evolution of the “disparate impact” and “disparate treatment” terminology, 
and the types of discrimination they are associated with). 

187 See Barocas & Selbst, supra note 8, at 694-714 (noting the ways in which algorithmic data 
mining techniques can lead to unintentional discrimination against historically prejudiced groups). 

188 Id. at 695. 

189 Id. at 700. 
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can escape disparate impact liability because the factors used are job-related?190 
If, as noted in subsection III.B.2, knowledge of class membership can be used 
to improve the fairness of outcomes for members of all classes, should doing 
so be considered disparate treatment? 

These doctrines were recently considered in Ricci v. DeStefano, in which 
the Supreme Court held that “before an employer can engage in intentional 
discrimination for the asserted purpose of avoiding or remedying an 
unintentional disparate impact, the employer must have a strong basis in 
evidence to believe it will be subject to disparate-impact liability if it fails to 
take the race-conscious, discriminatory action.”!%1 At issue was the City of 
New Haven’s test for firefighter promotions; though the test had been 
constructed in an attempt to ensure there was no discrimination by race,192 
the pass rates for minorities were about half of the pass rate for whites.193 The 
New Haven Civil Service Board did not certify the results of the test (and 
validate the promotions) due to concerns about fairness and disparate impact 
liability for the City.194 

Ricci demonstrates the tension between disparate treatment and disparate 
impact. Facially neutral policies can produce unequal results for protected 
classes, but remedying that disparate impact would require the state to treat 
people differently based on class membership, which Ricci forbids. Ricci also 
hints at the difficulties in squaring the Court’s move toward a colorblind 
interpretation of the Equal Protection Clause and the doctrine of disparate 
impact. The holding does not directly address the constitutional issue, but 
Justice Scalia’s concurrence does note that the “war between disparate impact 
and equal protection will be waged sooner or later.”195 Both of these doctrinal 
tensions are of concern to lawmakers and policymakers. 


2. Ricci Impels Designing for Nondiscrimination 


Although Ricci has generated wide-ranging conversation about equal 
protection, disparate treatment, and disparate impact, we wish to emphasize 
its implications for the governance of decision algorithms for processes where 
nondiscrimination is a goal. The holding in Ricci suggests that we cannot 
solely rely on auditing for legal reasons in addition to the reasons discussed 
in Section II.B. If an agency runs an algorithm that has a disparate impact, 
correcting those results after the fact will trigger the same kind of analysis as 


190 Id. at 707. 

191 557 U.S. 55, 585 (2009). 

192 Id. at 565. 

193 Id. at 586-87. 

194 Id. at 579. 

195 Id. at 595-96 (Scalia, J., concurring). 
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New Haven’s rejection of its firefighter test results. It is even possible that 
the Court will “subject some range of disparate impact compliance efforts to 
strict scrutiny,”196 a high bar that will be difficult to satisfy in most cases. 

The legal difficulties with correcting discriminatory algorithms ex post 
make measures to design algorithms for nondiscrimination even more 
important. The Court in Ricci took no issue with New Haven’s process of 
designing the test with an eye toward nondiscrimination, reasoning that “Title 
VII does not prohibit an employer from considering, before administering a test 
or practice, how to design that test or practice in order to provide a fair 
opportunity for all individuals, regardless of their race.”197 However, “once 
that process has been established and employers have made clear their 
selection criteria, they may not then invalidate the test results, thus upsetting 
an employee’s legitimate expectation not to be judged on the basis of race.”198 

The uneasy fit of algorithmic decisionmaking into the disparate treatment/ 
disparate impact framework does mean that someone could allege disparate 
treatment because the design of the algorithm includes inputs that are a proxy 
for class membership, resulting in a formal application of different rules to 
different groups of people. However, such a claim would be valid against 
virtually any system with a significant number of inputs. It seems more likely 
that courts would reject the formal-rule subset of disparate treatment for 
algorithmic decisions than that they would hold the majority of algorithmic 
decisionmaking to constitute disparate treatment. In the end, incorporating 
nondiscrimination in the initial design of algorithms is the safest path that 
decisionmakers can take, and we should encourage the development and 
deployment of technical tools to aid in that design. 


IV. FOSTERING COLLABORATION ACROSS COMPUTER SCIENCE, 
LAW, AND POLICY 


In this Part, we consider how the types of technological assurance described 
in previous Parts relate to mechanisms of oversight in law and public policy. In 
technical approaches, it is traditional to have a detailed, well-defined specification 
of the behavior of a system for all types of situations. In lawmaking and the 
application of public policy, it is normal, and even encouraged, for rules to be 
left open to interpretation, with details filled by human judgment emerging 
from disputes in specific cases that are resolved after the fact. We offer 
recommendations for dealing with this apparent mismatch, arguing for 
greater collaboration between experts in these two different fields. 


196 The Supreme Court, 2008 Term—Leading Cases, supra note 184, at 290. 
197 Ricci, 557 U.S. at 585. 
198 Id. 
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We emphasize that computer scientists cannot assume that the policy 
process will give them a meaningful, universal, and self-consistent theory of 
fairness to use as a specification for algorithms. There are structural, political, 
and jurisprudential reasons why no such theory exists today. Likewise, the 
policy process would likely not accept such a theory if it were generated by 
computer scientists. 

At the same time, lawmakers and policymakers will need to adapt in light 
of these new technologies. We highlight changes that stem from automated 
decisionmaking. First, choices made when designing computer systems 
embed specific policy decisions and values in those systems whether or not 
they provide for accountability. Algorithms can, nevertheless, permit direct 
accountability to the public or to other third parties, despite the fact that full 
transparency is neither sufficient nor always necessary for accountability. For 
both groups, we note that the interplay between these areas will raise new 
questions and may generate new insights into what the goals of these 
decisionmaking processes should be. 


A. Recommendations for Computer Scientists: Design for After-the-Fact Oversight 


Computer scientists may tend to think of accountability in terms of 
compliance with a detailed specification set forth before the creation of an 
algorithm. For example, it is typical for programmers to define bugs based on 
the specification for a program—anything that differs from the specification 
is a bug; anything that follows it is a feature.199 

This Section is intended to inform computer scientists that no one will 
remove all of the ambiguities and offer them a clear, complete specification. 
Although lawmakers and policymakers can offer clarifications or other changes 
to guide the work done by developers,200 drafters may be unable to remove 
certain ambiguities for political reasons or be unwilling to resolve details to 
meet flexibility objectives. As such, computer scientists must account for the 
lack of precision—and the corresponding need for after-the-fact oversight by 
courts or other reviewers—when designing decisionmaking algorithms. 

A computer scientist’s mindset can conflict deeply with many sources of 
authority to which developers may be responsible. Public opinion and social 


199 See, e.g, Michael Dubakov, Visual Specifications, MEDIUM (Oct. 26, 2013), https://medium. 
com/@mdubakov/visual-specifications-1d57822a485f [https://perma.cc/SE46-6B2C] (“No specs? No 
bugs.”); SF, What Is the Difference Between Bug and New Feature in Terms of Segregation of 
Responsibilities?, STACKEXCHANGE (July 12, 2011, 6:51 AM), http://programmers.stackexchange.com/ 
questions/92081/what-is-the-difference-between-bug-and-new-feature-in-terms-of-segregation-of-re 
[https://perma.cc/PPM6-HFAA] (“You could put an artificial barrier: if it’s against specs, it’s a bug. 
If it requires changing specs . . . it’s a feature.”). 

200 See infra Section IV.B. 
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norms are inherently not precisely specified. The corporate requirements to 
satisfy one’s supervisor (or one’s supervisor’s supervisor) may not be clear. 
Perhaps most importantly and least intuitively for computer scientists, the 
operations of U.S. law and public policy also work against clear specifications. 
These processes often deliberately create ambiguous laws and guidance, leaving 
details—or sometimes even major concepts—open to interpretation.201 

One cause of this ambiguity is the political reality of legislation. 
Legislators may be unable to gather majority support to agree on the details 
of a proposed law, but may be able to get a majority of votes to pass relatively 
vague language that leaves various terms and conditions unspecified.202 For 
example, different legislators may support conflicting specific proposals that 
can be encompassed by a more general bill.203 Even legislators who do not 
know precisely what they want may still object to a particular proposed detail; 
each detail that caused sufficient objections would need to be stripped out of 
a bill before it could become law. 

Another explanation of ambiguity is that legislators may have uncertainty 
about the situations to which a law or policy will apply. Drafters may worry 
that they have not fully considered all of the possibilities. This creates an 
incentive to build in enough flexibility to cover unexpected circumstances 
that currently exist or may exist in the future.204 The U.S. Constitution is 
often held up as a model in this regard: generalized provisions for governance 
and individual rights continue to be applicable even as the landscape of 
society changes dramatically.20s 

Finally, ambiguity may stem from shared uncertainty about how best to 
solve even a known problem. Here, drafters may feel that they know what 


201 See, e.g., Marbury v. Madison, 5 U.S. (1 Cranch) 137 (1803) (establishing the practice of 
judicial review, on which the Constitution was silent); see also 47 U.S.C. § 222(c)(1) (2012) (requiring 
a telecommunications carrier to get the “approval of the customer” to use or disclose customer 
proprietary network information, and requiring the Federal Communications Commission to define 
“approval”). 

202 See Victoria F. Nourse & Jane S. Schacter, The Politics of Legislative Drafting: A Congressional 
Case Study, 77 N.Y.U. L. REV. 575, 593 (2002) (“Several staffers thought that pressures of time, and 
the political imperative to get a bill ‘done, bred ambiguity. Indeed, one staffer emphasized that while 
it was well and good to draft a bill clearly, there was no guarantee that the clear language would be 
passed by the House or make it through conference.”). 

203 See Richard L. Hasen, Vote Buying, 88 CALIF. L. REV. 1323, 1339 (2000) (describing the 
practice of “legislative logrolling”). 

204 See, eg, 17 U.S.C. § 1201 (2012) (granting the Copyright Office the power to create 
exemptions from the statute’s prohibition on anti-circumvention). 

205 See DAVID A. STRAUSS, THE LIVING CONSTITUTION (2010). Laws governing law 
enforcement access to personal electronic records are often cited as a counterexample, with over- 
specific provisions in the Electronic Communications Privacy Act, 18 U.S.C. §§ 2510-2704 (2012), 
that fail to account for a shift in technology to a regime where most records reside with third party 
service providers, not users’ own computers. For a more detailed explanation, see Orin S. Kerr, 


Applying the Fourth Amendment to the Internet: A General Approach, 62 STAN. L. REV. 1005 (2010). 
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situations will arise but still not know how they want to deal with them. They 
may, in effect, choose to delegate authority to other parties by underspecifying 
particular aspects of a law or policy. Vagueness supports experimentation to 
help determine what methods are most effective or desirable.206 

The United States has a long history of dealing with these ambiguities 
through after-the-fact and retroactive oversight by the courts.207 In our 
common law system, ambiguities and uncertainties are left unaddressed until 
there is a dispute and their resolution becomes necessary. Disagreements 
about the application of a law or regulation to a specific set of facts are 
resolved through cases, and the areas of ambiguity are clarified over time by 
the accretion of many rulings on specific situations.208 Even when statutes and 
regulations may have specific and detailed language, they are interpreted 
through cases—with extensive deference often given to the expertise of 
administrative agencies.209 Those cases form binding precedents that, in the 
U.S. common law system, are an additional source of legal authority alongside 
the statutes themselves.2!0 The gradual development and extension of law and 
regulations through cases with specific fact patterns allows for careful 
consideration of meaning and effects at a level of granularity that is usually 
impossible to reach during the drafting process.21! 

In practice, these characteristics imply that computer scientists should 
focus on creating algorithms that are reviewable, not just compliant with the 
specifications that are generated in the drafting process.212 For example, this 
means it would have been good for the Diversity Visa Lottery described in 
Section ILE to use an algorithm that made fair, random choices and it would 
be desirable for the State Department to be able to demonstrate that property 
to a court or a skeptical lottery participant.2%3 


206 A similar logic—policy experimentation among the states—is one of the principles 
underlying federalism. See New State Ice Co. v. Liebmann, 285 U.S. 262, 311 (1932) (Brandeis, J., 
dissenting) (praising the ability of a state to “serve as a laboratory” for democracy). 

207 See generally E. ALLAN FARNSWORTH, AN INTRODUCTION TO THE LEGAL SYSTEM OF 
THE UNITED STATES (Steve Sheppard ed., 4th ed. 2010). 

208 See generally id. 

209 See Chevron U.S.A. Inc. v. Nat. Res. Def. Council, Inc., 476 U.S. 837 (1984). 

210 See generally FARNSWORTH, supra note 207. 

211 Id. 

212 Another possible conclusion is that certain algorithms should also be developed to be 
flexible, permitting adaptation as new cases, laws, or regulations add to the initial specifications. The 
need to adapt algorithms is discussed further in subsection IV.B.1. This also reflects the current 
insufficiency of building a system in accord with a particular specification, though oversight or 
enforcement bodies evaluating the decision at a later point in time will still need to be able to certify 
compliance with any actual specifications. 

213 Algorithms offer a new opportunity for decisionmaking processes to be reviewed by 
nontraditional overseers: decision recipients, members of the public, or even concerned 
nongovernmental organizations. We discuss this possibility further in subsection IV.B.2. 
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The technical approaches described in this Article214 provide several ways 
for algorithm designers to ensure that the actual basis for a decision can be 
verified later. With these tools, reviewers can check whether an algorithm 
actually was used to make a particular decision,215 whether random inputs 
were chosen fairly,2! and whether the algorithm comports with certain 
principles specified at the time of the design.217 Essentially, these technical 
tools allow continued after-the-fact evaluations of algorithms by allowing for 
and assisting the judicial system’s traditional role in ultimately determining 
the legality of particular decisionmaking.218 

Implementing the approaches described in this Article would improve the 
accountability of decisionmaking algorithms dramatically, but we see that 
implementation as only a first step. We encourage research into extensions of 
these technical tools, as well as new techniques designed to facilitate oversight. 


B. Recommendations for Lawmakers and Policymakers 


The other side of the coin is that lawmakers and policymakers need to 
recognize and adapt to the changes wrought by algorithmic decisionmaking. 
Characteristics of algorithms offer both new opportunities and new challenges 
for the development of legal regimes governing decisionmaking: algorithmic 
decisionmaking can reduce the benefits of ambiguity, increase accountability to 
the public, and permit greater accountability than was previously possible in 
cases where aspects of the decision process remain secret. 


1. Reduced Benefits of Ambiguity 


Although computer scientists can build algorithms to permit after-the-fact 
assessment and accountability, they cannot alter the fact that any algorithm 
design will encode specific values and involve specific rules. Furthermore, the 
design of a computer system may limit opportunities for after-the-fact 
accountability. In other words, if a system is not designed to permit certification 
of a particular characteristic, an oversight body cannot be certain that it will be 
able to certify that characteristic. Both of these traits imply that automated 
decisionmaking can exacerbate certain disadvantages of legal ambiguities. 

In the framework set forth above,2!9 we identify key drivers of ambiguity: 
political stalemate, uncertainty about future circumstances, and desire for 


214 See supra Sections II.B, III.B. 

215 See supra Section II.C. 

216 See supra subsection II.C.4. 

217 See supra subsection I1.C.1. 

218 Computer scientists model this after-the-fact input as an “oracle” that can be consulted only 
rarely on the acceptability of the algorithm. See Kroll, supra note 118. 

219 See supra Section IV.A. 
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policy experimentation. Here, with respect to each of these drivers, we will 
discuss how the shift to algorithmic decisionmaking diminishes the appeal of 
ambiguity, and we will suggest ways of retaining the functional benefits that 
ambiguity provides in the U.S. lawmaking system and ways that are more 
amenable to automation. 

Ambiguity stemming from political stalemate essentially passes the buck 
for determining details from legislators to someone later on in the process. 
These later actors tend to be more sheltered from political pressures and thus 
able to make specific decisions without risking their jobs at the next election. 
Judges and administrative agencies frequently fill this role. Courts are expected 
to offer impartial decisions resistant to public pressure,220 and administrative 
agencies are expected to retain staff who offer subject-matter expertise beyond 
what is expected of legislators, despite changes in political administrations.221 

However, this transfer of responsibility often works in less than ideal ways 
when it comes to software systems.?22 Fully automated decisionmaking may 
exacerbate these problems by adding another actor to whom the responsibility 
can devolve: the developer who programs the decisionmaking software. 
Citron offers examples of failures in automated systems that determine 
benefits eligibility, notably the airport “No Fly” lists, terrorist identifications, 
and punishment for “dead-beat” parents.223 Lawmakers should consider this 
possibility and avoid giving the responsibility for filling in the details of the law 
to program developers because (1) the algorithms will apply broadly, affecting 
all participants; (2) the program developer is unlikely to be held accountable 
by the current political process; and (3) the program developer is unlikely to 
have substantive expertise about the political decision being made.224 


220 See, e.g., THE FEDERALIST NO. 78 (Alexander Hamilton) (laying out the philosophy that 
the judiciary’s role is to secure an “impartial administration of the laws”). However, the rise of 
elected judges raises questions about this traditional role of the court system. See Stephen J. Choi 
et al., Professionals or Politicians: The Uncertain Empirical Case for an Elected Rather Than Appointed 
Judiciary (Univ. of Chi. Law Sch., John M. Olin Law & Economics Working Paper No. 357, 2007) 
(finding that elected judges behave more like politicians than appointed independent judges). 

221 ‘This is the rationale of the Chevron doctrine of judicial deference to administrative agency 
actions. Chevron U.S.A. Inc. v. Nat. Res. Def. Council, Inc., 476 U.S. 837 (1984). 

222 For example, Citron argues that “[d]istortions in policy have been attributed to the fact 
that programmers lack ‘policy knowledge,” and that this leads to software that does not reflect policy 
goals, Citron, supra note 6, at 1261. Ohm also reports on a comment of Felten that “[i]n technology 
policy debates, lawyers put too much faith in technical solutions, while technologists put too much 
faith in legal solutions.” Paul Ohm, Breaking Felten’s Third Law: How Not to Fix the Internet, 87 DENV. 
L. REV. ONLINE (2010), http://www.denverlawreview.org/how-to-regulate/2010/2/22/breaking- 
feltens-third-law-how-not-to-fix-the-internet.html  [https://perma.cc/6bRGQ-KUMW] (internal 
quotation marks omitted). 

223 Citron, supra note 6, at 1256-57. 

224 ld. at 1254-55. A distinction should be drawn here between the responsibilities given to 
individual developers of particular algorithms and the responsibilities given to computer scientists 
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One potential method for restricting the discretion of developers without 
requiring specifications in the legislation itself would be for administrative 
agencies to publish guidance for software development. Difficulties in 
translating between code choices and policy effects still would exist, but they 
could be partly eased using the technical methods we have described.225 For 
example, administrative agencies could work together with developers to 
identify the properties they want a piece of software to possess, and the 
program could then be designed to satisfy those properties and permit proof. 

Ambiguity generated by uncertainty about the situational circumstances 
or ambiguity motivated by a desire for policy experimentation presents a 
more difficult concern. Here, the problem raised by automated decisionmaking 
is that a piece of software locks in a particular interpretation of law for the 
duration of its use, and, especially in government contexts, provisions to 
update the software code may not be made. Worries about changing or 
unexpected circumstances could be assuaged by adding sunset provisions to 
software systems,226 requiring periodic review and reconsideration of the 
software. Additionally, software should be designed with eventual revisions and 
updates in mind. As for preserving the benefits of policy experimentation, the 
traditional solution might be having multiple programs that take multiple 
approaches deployed simultaneously. A more sophisticated version of this 
solution is the incorporation of machine learning into decisionmaking 
systems. Again, machine learning can have its own fairness pitfalls,227 and 
care should be taken to consider fair machine learning methods228 and to build 
in precautions like persistent testing of the hypotheses built into the machine 
learning model.229 

More generally, the benefits of ambiguity decrease in the case of 
algorithmic decisionmaking. Here, an uninformed programming actor may 
determine the details and then apply them broadly. In addition, the choice of 
algorithm cements the particular policy choices encoded in that software for 


in general. Great gains can be made by improved dialogue between computer scientists and 
lawmakers and policymakers about how to design algorithms to reach social goals. 

225 See supra Sections II.B, III.B. 

226 ‘The effectiveness of sunset provisions in leading to actual reconsideration and change is 
debatable. The inertia of the pre-existing choices can be hard to overcome. See, e.g., Mark A. Lemley 
& David McGowan, Legal Implications of Network Economic Effects, 86 CALIF. L. REV. 479, 481-82 
(1998) (noting that stare decisis, confusion regarding the role of theory, differing normative values, 
and other factors impede the progress of the law). 

227 See supra subsection III.B.1 (noting that machine learning programs give predictions but 
not confidence levels). 

228 See supra Sections III.A-B. 

229 In other words, even after a machine learning algorithm determines that a particular rule 
should be used to produce particular results, it always should continue to test inputs that do not 
follow that rule. See, e.g., RUSSELL & NORVIG, supra note 67. 
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as long as it is used. Drafters should instead consider whether they should 
increase the specificity offered by law and policy governing these algorithms 
to prevent coders from filling the ambiguity. 

To a certain extent, this question mirrors the rules versus standards debate 
about the relative merits of laws that specify actions and their repercussions 
(for example, a speed limit) and those that espouse a principle open to 
interpretation (for example, “drive at a speed reasonable for the conditions”).230 
Rules give clarity and forewarning, while standards offer greater flexibility 
for interpretation.231 

Here, the question is whether drafters should include additional and 
clearer specifications for developers. In practice, drafters may wish to 
incorporate a set of narrow rules within a broad, overarching standard. For 
example, drafters could include specifications of each of the properties that 
they want a piece of software to possess and requirements that the developer 
design that program in a way that renders those properties provable upon 
review. Additionally, drafters might consider requiring a general statement of 
purpose for the algorithm. Doing so would give the developer some flexibility 
in writing the code while also ensuring that particular properties can be 
checked later. 


2. Accountability to the Public 


Oversight is traditionally performed by courts, enforcement agencies, or 
other designated entities such as government prosecutors.232 Typically, the 
public and third parties have an indirect oversight role through the ability to 
provide political feedback and the ability to bring lawsuits if their specific 
circumstances allow.233 The use of software can alter how effectively the legal 
system and the public can oversee the decisionmaking process. 

In one sense, decisionmaking computer systems can enhance accountability 
to the public and interested third parties by permitting greater involvement 
in oversight. The technical tools we describe allow for a more direct form of 
oversight by these parties. Unlike traditional legal oversight mechanisms that 
generally require discovery or the gathering of internal evidence, the technical 
tools may enable verifications by the public and by third parties that are 


230 See, e.g., Louis Kaplow, Rules Versus Standards: An Economic Analysis, 42 DUKE L.J. 557, 562- 
66 (1992) (arguing that rules are more costly to promulgate while standards are more costly on 
individuals). 

231 See Kathleen M. Sullivan, The Supreme Court, 1991 Term—Forward: The Justices of Rules and 
Standards, 106 HARV. L. REV. 22, 26 (1992) (explaining the rule versus standard choice in terms of 
force of precedent, constitutional reading, and formulation of operative tests). 

232 See FARNSWORTH, supra note 207. 

233 The public can vote political leaders out of office and aggrieved parties can bring lawsuits 
to seek vindication. 
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completely independent from the organizations using the software. For 
example, technologically proficient members of the public or third parties 
could verify that a particular algorithm was used in a program or that the 
program has particular properties. In addition, a system could be built to 
enable participants to check these properties for their own outcomes so that 
nontechnical users could verify these facts while the system as a whole would 
be overseen by others—potentially both inside and outside of government— 
who have the necessary technological expertise. As another example, third 
parties could be involved in generating fair randomness.234 

In contrast to the possibility for enhanced public accountability, the use 
of software without the reliance on technical tools for oversight, as we have 
described, can reduce accountability to the public because courts and other 
policy actors are generally ill-equipped to evaluate software, thereby 
hampering our traditional scrutiny of decisionmaking. The U.S. court system 
is designed to protect against wrongful government actions through the 
power of judicial review.235 Judicial review gives judges the power and 
responsibility to determine if government actions comply with legal 
obligations. Similarly, for private actions, the legal system vests judges and 
regulatory agencies with the authority to determine whether those actions are 
consistent with legal standards. 

The use of software systems to make decisions, however, shifts these 
burdens to external experts or to the organizations creating and deploying the 
software. Courts and enforcement agencies are no longer able to make a 
determination as to whether the rules have been properly applied or whether 
fairness obligations have been met. That determination shifts to the experts 
evaluating the automated decisionmaking process. One way to address this 
unintended shift in responsibility is to appoint technical experts as special 
masters. Courts typically appoint special masters to perform functions on 
behalf of the court that require special skill or knowledge.236 

Another issue that challenges public accountability is the validation of the 
technical tools we have described. For courts, technical tools cannot be 
accepted until their integrity and reliability are proven. Courts have long 
confronted the problem of the admissibility of scientific evidence. There is a 
rich literature about the standards courts should use to admit expert scientific 


234 See supra note 116 (using a quantum source to generate randomness). 

235 See Marbury v. Madison, 5 U.S. (1 Cranch) 137, 177 (1803) (“It is emphatically the province 
and duty of the judicial department to say what the law is.”). 

236 See, e.g., United States v. Microsoft Corp., 147 F.3d 935, 959 n.4 (D.C. Cir. 1998) (noting 
Larry Lessig’s role as a court-appointed special master for technical issues in the antitrust case 
brought against Microsoft). 
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evidence, and, even now, federal and state standards vary.237 The courts, for 
example, took years during the 1980s and gos to establish and accept the 
scientific validity of DNA and the methods used to isolate and test DNA.238 
Even now, there are concerns that some scientific methods (e.g. matching 
DNA based on mixtures of DNA) may be receiving undeserved deference 
from courts and thus resulting in faulty findings of fact.239 Following much 
debate, the Federal Rules of Evidence spelled out a federal standard for the 
acceptability of new scientific methods in adversarial proceedings.240 In 1993, 
the Supreme Court adjusted those standards to take account of factors that 
include testing, peer review and publication.241 These evidentiary standards 
address the validation of technical tools used to examine automated 
decisionmaking, but still leave open the assurance of the technical tools’ 
reliability. Ordinarily, the U.S. legal system relies on the adversarial process 
to assure the accuracy of findings. This attribute may be preserved by 
allowing multiple experts to test software-driven processes. 


3. Secrets and Accountability 


Implementing automated decisionmaking in a socially and politically 
acceptable way requires progress in our ability to communicate and understand 
fine-grained partial information about how decisions are reached. Full 
transparency (disclosing everything) is technically trivial but politically and 
practically infeasible and may not be useful, as described in Section ILA. 
However, disclosing nothing about the basis for a decision is socially 
unacceptable and generally poses a technical challenge. Lawmakers and 


237 See, e.g., Paul C. Giannelli, The Admissibility of Novel Scientific Evidence: Frye v. United 
States, a Half-Century Later, 80 COLUM. L. REV. 1197 (1980) (highlighting the development of the 
standards used for evidentiary scientific evidence); Heather G. Hamilton, Note, The Movement from 
Frye to Daubert: Where Do the States Stand?, 38 JURIMETRICS 201 (1998) (emphasizing the lack of 
uniformity of state approaches). 

238 See, e.g., NAT'L RESEARCH COUNCIL, THE EVALUATION OF FORENSIC DNA EVIDENCE 
166-211 (1996) (discussing the legal implications of the use of forensic DNA testing as well as the 
procedural and evidentiary rules that affect such use). 

239 Logan Koepke, Should Secret Code Help Convict?, MEDIUM (Mar. 24, 2016), https://medium. 
com/equal-future/should-secret-code-help-convict-7c864baffe1s#.jgkicwhoo _ [https://perma.cc/6LNW- 
WN6W]. 

240 See FED. R. EVID. 702 (“A witness who is qualified as an expert by knowledge, skill, 


experience, training, or education may testify in the form of an opinion or otherwise if ... the 
expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand 
the evidence or to determine a fact in issue... .”) 


241 See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 592-95 (1993) (explaining that a 
judge faced with a proffer of expert scientific testimony must assess whether the testimony’s 
underlying reasoning is valid, and in doing so, consider whether the technique or theory in question 
can be tested and whether it has been subjected to peer review and publication). 
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policymakers should remember that it is possible to make an algorithm 
accountable without the evaluator having full access to the algorithm.242 

U.S. law and policy often focus on transparency and sometimes even 
equate oversight with transparency for the overseer.243 As such, accountability 
without full transparency may seem counterintuitive. However, oversight 
based on partial information occurs regularly within the legal system. Courts 
prevent consideration of many types of information for various policy reasons: 
disclosures of classified information may be prevented or limited to preserve 
national security;244 juvenile records may be sealed because of the notion that 
mistakes made in one’s youth should not follow them forever;245 and other 
evidence is deemed inadmissible for a multitude of reasons, including being 
unscientific,246 hearsay,247 inflammatory,?48 or illegally obtained.249 Thus, all of 
the rules of evidence could be construed as precedent for the idea that optimal 
oversight does not require full information. 

There are strong policy justifications for holding back information in the 
case of automated decisionmaking. Revealing software source code and input 
data can expose trade secrets, violate privacy, hamper law enforcement, or 
lead to gaming of the decisionmaking process.250 The advantage of computer 
systems is that concealment of code and data does not imply an inability to 
analyze the code and data. The technical tools we describe give lawmakers 
and policymakers the ability to keep software programs and their inputs 
secret while still rendering them accountable. They can implement these tools 
in government-run algorithms, such as the DVL, and incentivize 
nongovernmental actors to use them, perhaps by mandating use or by 
requiring transparency —at least to courts —of code and inputs if they do not 
employ such technical tools. 


242 See supra subsections II.C.3-I.D. 

243 See, e.g., 5 U.S.C. § 552 (2012) (requiring agencies to make government records available to 
the public); 15 U.S.C. § 6803 (2012) (requiring financial institutions to provide annual privacy 
notices to customers as a transparency measure). 

244 See 18 U.S.C. § 798(a) (2012) (providing that the disclosure of classified government 
information may result in criminal liability). 

245 See, e.g., N.Y. CRIM. PROC. § 720.15 (requiring filing under seal in juvenile proceedings). 

246 See FED. R. EVID. 702 (establishing the court’s discretion to admit scientific evidence). 

247 See FED. R. EVID. 802 (stating that hearsay evidence is inadmissible unless a federal 
statute, the rules of evidence, or the Supreme Court provides otherwise). 

248 See FED. R. EVID. 403 (providing for the exclusion of relevant evidence for prejudice). 

249 See, e.g., 18 U.S.C. § 2515 (2012) (setting an exclusionary rule for evidence obtained through 
wire tap or interception). 


250 See Sections I].A-B. 


Big Data’s Disparate Impact 


Solon Barocas* & Andrew D. Selbst** 


Advocates of algorithmic techniques like data mining argue that 
these techniques eliminate human biases from the decision-making 
process. But an algorithm is only as good as the data it works with. 
Data is frequently imperfect in ways that allow these algorithms to 
inherit the prejudices of prior decision makers. In other cases, data 
may simply reflect the widespread biases that persist in society at 
large. In still others, data mining can discover surprisingly useful 
regularities that are really just preexisting patterns of exclusion and 
inequality. Unthinking reliance on data mining can deny historically 
disadvantaged and vulnerable groups full participation in society. 
Worse still, because the resulting discrimination is almost always an 
unintentional emergent property of the algorithm’s use rather than a 
conscious choice by its programmers, it can be unusually hard to 
identify the source of the problem or to explain it to a court. 


This Essay examines these concerns through the lens of 
American antidiscrimination law—more particularly, through Title 
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VII's prohibition of discrimination in employment. In the absence of 
a demonstrable intent to discriminate, the best doctrinal hope for 
data mining’s victims would seem to lie in disparate impact doctrine. 
Case law and the Equal Employment Opportunity Commission’s 
Uniform Guidelines, though, hold that a practice can be justified as a 
business necessity when its outcomes are predictive of future 
employment outcomes, and data mining is specifically designed to 
find such statistical correlations. Unless there is a reasonably 
practical way to demonstrate that these discoveries are spurious, 
Title VII would appear to bless its use, even though the correlations it 
discovers will often reflect historic patterns of prejudice, others’ 
discrimination against members of protected groups, or flaws in the 
underlying data. 


Addressing the sources of this unintentional discrimination and 
remedying the corresponding deficiencies in the law will be difficult 
technically, difficult legally, and difficult politically. There are a 
number of practical limits to what can be accomplished 
computationally. For example, when discrimination occurs because 
the data being mined is itself a result of past intentional 
discrimination, there is frequently no obvious method to adjust 
historical data to rid it of this taint. Corrective measures that alter 
the results of the data mining after it is complete would tread on 
legally and politically disputed terrain. These challenges for reform 
throw into stark relief the tension between the two major theories 
underlying _antidiscrimination law:  anticlassification and 
antisubordination. Finding a solution to big data’s disparate impact 
will require more than best efforts to stamp out prejudice and bias; it 
will require a wholesale reexamination of the meanings of 
“discrimination” and “fairness.” 
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INTRODUCTION 


“Big Data” is the buzzword of the decade.’ Advertisers want data to reach 
profitable consumers,’ medical professionals to find side effects of prescription 
drugs,’ supply-chain operators to optimize their delivery routes,’ police to 
determine where to focus resources, and social scientists to study human 
interactions.° Though useful, however, data is not a panacea. Where data is 
used predictively to assist decision making, it can affect the fortunes of whole 
classes of people in consistently unfavorable ways. Sorting and selecting for 
the best or most profitable candidates means generating a model with winners 
and losers. If data miners are not careful, the process can result in 
disproportionately adverse outcomes concentrated within historically 
disadvantaged groups in ways that look a lot like discrimination. 

Although we live in the post—civil rights era, discrimination persists in 
American society and is stubbornly pervasive in employment, housing, credit, 
and consumer markets.’ While discrimination certainly endures in part due to 
decision makers’ prejudices, a great deal of modern-day inequality can be 
attributed to what sociologists call “institutional” discrimination.* Unconscious, 
implicit biases and inertia within society’s institutions, rather than intentional 


1. Contra Sanjeev Sardana, Big Data: It’s Not a Buzzword, It’s a Movement, FORBES (Nov. 
20, 2013), http:/www.forbes.com/sites/sanjeevsardana/2013/11/20/bigdata [https://perma.cc/9 Y37- 
ZFT5]. 

2. Tanzina Vega, New Ways Marketers Are Manipulating Data to Influence You, N.Y. 
TIMES: Bits (June 19, 2013, 9:49 PM), http://bits.blogs.nytimes.com/2013/06/19/new-ways- 
marketers-are-manipulating-data-to-influence-you [https://perma.cc/238F-9T8X]. 

3. Nell Greenfieldboyce, Big Data Peeps at Your Medical Records to Find Drug Problems, 
NPR (July 21, 2014, 5:15 AM), http:/;www.npr.org/blogs/health/2014/07/21/332290342/big-data- 
peeps-at-your-medical-records-to-find-drug-problems [https://perma.cc/GMT4-ECBD]. 

4. Business by Numbers, ECONOMIST (Sept. 13, 2007), 
http:/Avww.economist.com/node/9795 140 [https://perma.cc/7YC2-DMYA]. 

5. Nadya Labi, Misfortune Teller, ATLANTIC (Jan—Feb. 2012), http://www.theatlantic.com/ 
magazine/archive/2012/01/misfortune-teller/308846 [https://perma.cc/7L72-J5L9]. 

6. David Lazer et al., Computational Social Science, 323 Sct. 721, 722 (2009). 

7. Devah Pager & Hana Shepherd, The Sociology of Discrimination: Racial Discrimination 
in Employment, Housing, Credit, and Consumer Markets, 34 ANN. REV. SOC. 181, 182 (2008). 

8. Id. 
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choices, account for a large part of the disparate effects observed.’ Approached 
without care, data mining can reproduce existing patterns of discrimination, 
inherit the prejudice of prior decision makers, or simply reflect the widespread 
biases that persist in society. It can even have the perverse result of 
exacerbating existing inequalities by suggesting that historically disadvantaged 
groups actually deserve less favorable treatment. 

Algorithms’ could exhibit these tendencies even if they have not been 
manually programmed to do so, whether on purpose or by accident. 
Discrimination may be an artifact of the data mining process itself, rather than 
a result of programmers assigning certain factors inappropriate weight. Such a 
possibility has gone unrecognized by most scholars and policy makers, who 
tend to fear concealed, nefarious intentions or the overlooked effects of human 
bias or error in hand coding algorithms.'' Because the discrimination at issue is 
unintentional, even honest attempts to certify the absence of prejudice on the 
part of those involved in the data mining process may wrongly confer the 
imprimatur of impartiality on the resulting decisions. Furthermore, because the 
mechanism through which data mining may disadvantage protected classes is 
less obvious in cases of unintentional discrimination, the injustice may be 
harder to identify and address. 

In May 2014, the White House released a report titled Big Data: Seizing 
Opportunities, Preserving Values (Podesta Report), which hinted at the 
discriminatory potential of big data.'* The report finds “that big data analytics 
have the potential to eclipse longstanding civil rights protections in how 
personal information is used in housing, credit, employment, health, education, 
and the marketplace.”'° It suggests that there may be unintended discriminatory 


9. See Andrew Grant-Thomas & john a. powell, Toward a Structural Racism Framework, 15 
POVERTY & RACE 3, 4 (“Institutional racism’ was the designation given in the late 1960s to the 
recognition that, at very least, racism need not be individualist, essentialist or intentional.”). 

10. An “algorithm” is a formally specified sequence of logical operations that provides step- 
by-step instructions for computers to act on data and thus automate decisions. SOLON BAROCAS ET 
AL., DATA & CIVIL RIGHTS: TECHNOLOGY PRIMER (2014), http://www.datacivilrights.org/pubs/2014- 
1030/Technology.pdf [https://perma.cc/X3 YX-XHNA]. Algorithms play a role in both automating the 
discovery of useful patterns in datasets and automating decision making that relies on these 
discoveries. This Essay uses the term to refer to the latter. 

11. See, e.g., Kate Crawford & Jason Schultz, Big Data and Due Process: Toward a 
Framework to Redress Predictive Privacy Harms, 55 B.C. L. REV. 93, 101 (2014) (“[H]ousing 
providers could design an algorithm to predict the [race, gender, or religion] of potential buyers or 
renters and advertise the properties only to those who [meet certain] profiles.”); Danielle Keats Citron 
& Frank Pasquale, The Scored Society: Due Process for Automated Predictions, 89 WASH. L. REV. 1, 
4 (2014) (“Because human beings program predictive algorithms, their biases and values are 
embedded into the software’s instructions. . . .”); Danielle Keats Citron, Technological Due Process, 
85 WASH. U. L. REV. 1249, 1254 (2008) (“Programmers routinely change the substance of rules when 
translating them from human language into computer code.”’). 

12. EXEC. OFFICE OF THE PRESIDENT, BIG DATA: SEIZING OPPORTUNITIES, PRESERVING 
VALUES (May 2014), http://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report 
_5.1.14 final _print.pdf [https://perma.cc/ZXB4-SDL9]. 

13. Id. (introductory letter). 
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effects from data mining but does not detail how they might come about. ' 
Because the origin of the discriminatory effects remains unexplored, the 
report’s approach does not address the full scope of the problem. 

The Podesta Report, as one might expect from the executive branch, seeks 
to address these effects primarily by finding new ways to enforce existing law. 
Regarding discrimination, the report primarily recommends that enforcement 
agencies, such as the Department of Justice, Federal Trade Commission, 
Consumer Financial Protection Bureau, and Equal Employment Opportunity 
Commission (EEOC), increase their technical expertise and “develop a plan for 
investigating and resolving violations of law in such cases.”"° 

As this Essay demonstrates, however, existing law largely fails to address 
the discrimination that can result from data mining. The argument is grounded 
in Title VII because, of all American antidiscrimination jurisprudence, Title 
VII has a particularly well-developed set of case law and scholarship. Further, 
there exists a rapidly emerging field of “work-force science,”'® for which Title 
VII will be the primary vehicle for regulation. Under Title VII, it turns out that 
some, if not most, instances of discriminatory data mining will not generate 
liability. While the Essay does not show this to be true outside of Title VII 
itself, the problem is likely not particular to Title VII. Rather, it is a feature of 
our current approach to antidiscrimination jurisprudence, with its focus on 
procedural fairness. The analysis will likely apply to other traditional areas of 
discrimination, such as housing or disability discrimination. Similar tendencies 
to disadvantage the disadvantaged will likely arise in areas that regulate 
legitimate economic discrimination, such as credit and insurance. 

This Essay proceeds in three Parts. Part I introduces the computer science 
literature and proceeds through the various steps of solving a problem with data 
mining: defining the target variable, labeling and collecting the training data, 
using feature selection, and making decisions on the basis of the resulting 
model. Each of these steps creates possibilities for a final result that has a 
disproportionately adverse impact on protected classes, whether by specifying 
the problem to be solved in ways that affect classes differently, failing to 
recognize or address statistical biases, reproducing past prejudice, or 
considering an insufficiently rich set of factors. Even in situations where data 
miners are extremely careful, they can still effect discriminatory results with 
models that, quite unintentionally, pick out proxy variables for protected 
classes. Finally, Part I notes that data mining poses the additional problem of 


14. Id. at 64 (“This combination of circumstances and technology raises difficult questions 
about how to ensure that discriminatory effects resulting from automated decision processes, whether 
intended or not, can be detected, measured, and redressed.”). 

15. Id. at 65. 

16. Steve Lohr, Big Data, Trying to Build Better Workers, N.Y. TIMES (Apr. 20, 2013), 
http:/Avww.nytimes.com/2013/04/2 1/technology/big-data-trying-to-build-better-workers.html 
[https://perma.cc/CEL2-P9XB]. 
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giving data miners the ability to disguise intentional discrimination as 
accidental. 

In Part II, the Essay reviews Title VII jurisprudence as it applies to data 
mining. Part II discusses both disparate treatment and disparate impact, 
examining which of the various data mining mechanisms identified in Part I 
will trigger liability under either Title VII theory. At first blush, either theory is 
viable. Disparate treatment is viable because data mining systems treat 
everyone differently; that is their purpose. Disparate impact is also viable 
because data mining can have various discriminatory effects, even without 
intent. But as Part II demonstrates, data mining combines some well-known 
problems in discrimination doctrines with new challenges particular to data 
mining systems, such that liability for discriminatory data mining will be hard 
to find. Part II concludes with a discussion of the new problems of proof that 
arise for intentional discrimination in this context. 

Finally, Part III addresses the difficulties reformers would face in 
addressing the deficiencies found in Part II. These difficulties take two forms: 
complications internal to the logic of data mining and political and 
constitutional difficulties external to the problem. Internally, the different steps 
in a data mining problem require constant subjective and fact-bound 
judgments, which do not lend themselves to general legislative resolution. 
Worse, many of these are normative judgments in disguise, about which there 
is not likely to be consensus. Externally, data mining will force society to 
explicitly rebalance the two justifications for antidiscrimination law—rooting 
out intentional discrimination and equalizing the status of historically 
disadvantaged communities. This is because methods of proof and corrective 
measures will often require an explicit commitment to substantive remediation 
rather than merely procedural remedies. In certain cases, data mining will make 
it simply impossible to rectify discriminatory results without engaging with the 
question of what level of substantive inequality is proper or acceptable in a 
given context. Given current political realities and trends in constitutional 
doctrines, legislation enacting a remedy that results from these discussions 
faces an uphill battle. To be sure, data mining also has the potential to help 
reduce discrimination by forcing decisions onto a more reliable empirical 
foundation and by formalizing decision-making processes, thus limiting the 
opportunity for individual bias to affect important assessments.'’ In many 
situations, the introduction of data mining will be a boon to civil rights, even 
where it fails to root out discrimination altogether, and such efforts should be 
encouraged. Yet, understanding when and why discrimination persists in cases 
of data-driven decision making reveals important and sometimes troubling 
limits to the promise of big data, for which there are no ready solutions. 


17. Tal Z. Zarsky, Automated Prediction: Perception, Law, and Policy, COMM. ACM, 
Sept. 2012, at 33-35. 
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I. 
How DATA MINING DISCRIMINATES 


Although commentators have ascribed myriad forms of discrimination to 
data mining,'* there remains significant confusion over the precise mechanisms 
that render data mining discriminatory. This Part develops a taxonomy that 
isolates and explicates the specific technical issues that can give rise to models 
whose use in decision making may have a disproportionately adverse impact on 
protected classes. By definition, data mining is always a form of statistical (and 
therefore seemingly rational) discrimination. Indeed, the very point of data 
mining is to provide a rational basis upon which to distinguish between 
individuals and to reliably confer to the individual the qualities possessed by 
those who seem statistically similar. Nevertheless, data mining holds the 
potential to unduly discount members of legally protected classes and to place 
them at systematic relative disadvantage. Unlike more subjective forms of 
decision making, data mining’s ill effects are often not traceable to human bias, 
conscious or unconscious. This Part describes five mechanisms by which these 
disproportionately adverse outcomes might occur, walking through a sequence 
of key steps in the overall data mining process. 


A. Defining the “Target Variable” and “Class Labels” 


In contrast to those traditional forms of data analysis that simply return 
records or summary statistics in response to a specific query, data mining 
attempts to locate statistical relationships in a dataset.'” In particular, it 
automates the process of discovering useful patterns, revealing regularities 
upon which subsequent decision making can rely. The accumulated set of 
discovered relationships is commonly called a “model,” and these models can 
be employed to automate the process of classifying entities or activities of 
interest, estimating the value of unobserved variables, or predicting future 
outcomes.” Familiar examples of such applications include spam or fraud 
detection, credit scoring, and insurance pricing. These examples all involve 
attempts to determine the status or likely outcome of cases under consideration 
based solely on access to correlated data.”' Data mining helps identify cases of 


18. Solon Barocas, Data Mining and the Discourse on Discrimination, PROC. DATA ETHICS 
WORKSHOP (2014), https://dataethics.github.io/proceedings/DataMiningandtheDiscourseOn 
Discrimination.pdf [https://perma.cc/D3LT-GS2X]. 

19. See generally Usama Fayyad, The Digital Physics of Data Mining, 44 COMM. ACM, Mar. 
2001, at 62. 

20. More formally, classification deals with discrete outcomes, estimation deals with 
continuous variables, and prediction deals with both discrete outcomes and continuous variables, but 
specifically for states or values in the future. MICHAEL J. A. BERRY & GORDON S. LINOFF, DATA 
MINING TECHNIQUES: FOR MARKETING, SALES, AND CUSTOMER RELATIONSHIP MANAGEMENT 8— 
11 (2004). 

21. Pedro Domingos, A Few Useful Things to Know About Machine Learning, COMM. ACM, 
Oct. 2012, at 78-80. 
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spam and fraud and anticipate default and poor health by treating these states 
and outcomes as a function of some other set of observed characteristics.” In 
particular, by exposing so-called “machine learning” algorithms to examples of 
the cases of interest (previously identified instances of fraud, spam, default, and 
poor health), the algorithm “learns” which related attributes or activities can 
serve as potential proxies for those qualities or outcomes of interest.”* 

Two concepts from the machine learning and data mining literature are 
important here: “target variables” and “class labels.” The outcomes of interest 
discussed above are known as target variables.” While the target variable 
defines what data miners are looking for, “class labels” divide all possible 
values of the target variable into mutually exclusive categories. 

The proper specification of the target variable is frequently not obvious, 
and the data miner’s task is to define it. To start, data miners must translate 
some amorphous problem into a question that can be expressed in more formal 
terms that computers can parse. In particular, data miners must determine how 
to solve the problem at hand by translating it into a question about the value of 
some target variable. The open-endedness that characterizes this part of the 
process is often described as the “art” of data mining. This initial step requires 
a data miner to “understand[] the project objectives and requirements from a 
business perspective [and] then convert[] this knowledge into a data mining 
problem definition.” Through this necessarily subjective process of 
translation, data miners may unintentionally parse the problem in such a way 
that happens to systematically disadvantage protected classes. 

Problem specification is not a wholly arbitrary process, however. Data 
mining can only address problems that lend themselves to formalization as 
questions about the state or value of the target variable. Data mining works 
exceedingly well for dealing with fraud and spam because these cases rely on 
extant, binary categories. A given instance either is or is not fraud or spam, and 
the definitions of fraud or spam are, for the most part, uncontroversial.”° A 
computer can then flag or refuse transactions or redirect emails according to 


22. Id. 

23. Id. 

24. COMM. ON THE ANALYSIS OF MASSIVE DATA ET AL., FRONTIERS IN MASSIVE DATA 
ANALYSIS 101 (2013), http://www.nap.edu/catalog.php?record_id=18374 [https://perma.cc/SDNQ- 
UFE4]. The machine learning community refers to classification, estimation, and prediction—the 
techniques that we discuss in this Essay—as “supervised” learning because analysts must actively 
specify a target variable of interest. Id. at 104. Other techniques known as “unsupervised” learning do 
not require any such target variables and instead search for general structures in the dataset, rather than 
patterns specifically related to some state or outcome. /d. at 102. Clustering is the most common 
example of “unsupervised” learning, in that clustering algorithms simply reveal apparent hot spots 
when plotting the data in some fashion. Jd. We limit the discussion to supervised learning because we 
are primarily concerned with the sorting, ranking, and predictions enabled by data mining. 

25. PETE CHAPMAN ET AL., CRISP-DM 1.0: STEP-BY-STEP DATA MINING GUIDE 10 (2000). 

26. See David J. Hand, Classifier Technology and the Illusion of Progress, 21 STAT. SCI. 1, 10 
(2006). 
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well-understood distinctions.”’ In these cases, data miners can simply rely on 
these simple, preexisting categories to define the class labels. 

Sometimes, though, defining the target variable involves the creation of 
new classes. Consider credit scoring, for instance. Although now taken for 
granted, the predicted likelihood of missing a certain number of loan 
repayments is not a self-evident answer to the question of how to successfully 
extend credit to consumers.” Unlike fraud or spam, “creditworthiness” is an 
artifact of the problem definition itself. There is no way to directly measure 
creditworthiness because the very notion of creditworthiness is a function of 
the particular way the credit industry has constructed the credit issuing and 
repayment system. That is, an individual’s ability to repay some minimum 
amount of an outstanding debt on a monthly basis is taken to be a nonarbitrary 
standard by which to determine in advance and all-at-once whether he is 
worthy of credit.” 

Data mining has many uses beyond spam detection, fraud detection, credit 
scoring, and insurance pricing. As discussed in the introduction, this Essay will 
focus on the use of data mining in employment decisions. Extending this 
discussion to employment, then, where employers turn to data mining to 
develop ways of improving and automating their search for good employees, 
they face a number of crucial choices. 

Like creditworthiness, the definition of a good employee is not a given. 
“Good” must be defined in ways that correspond to measurable outcomes: 
relatively higher sales, shorter production time, or longer tenure, for example. 
When employers mine data for good employees, they are, in fact, looking for 
employees whose observable characteristics suggest that they would meet or 
exceed some monthly sales threshold, perform some task in less than a certain 
amount of time, or remain in their positions for more than a set number of 
weeks or months. Rather than drawing categorical distinctions along these 
lines, data mining could also estimate or predict the specific numerical value of 
sales, production time, or tenure period, enabling employers to rank rather than 
simply sort employees. 

These may seem like eminently reasonable things for employers to want 
to predict, but they are, by necessity, only part of an array of possible 
definitions of “good.” An employer may instead attempt to define the target 
variable in a more holistic way—by, for example, relying on the grades that 
prior employees have received in annual reviews, which are supposed to reflect 


27. Though described as a matter of detection, this is really a classification task, where any 
given transaction or email can belong to one of two possible classes, respectively: fraud or not fraud, 
or spam or not spam. 

28. See generally Martha Ann Poon, What Lenders See—A History of the Fair Isaac 
Scorecard, (2013) (unpublished Ph.D. dissertation, University of California, San Diego), 
http://search.proquest.com/docview/1520318884 [https://perma.cc/YD3S-B9N7]. 

29. Hand, supra note 26, at 10. 
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an overall assessment of performance. These target variable definitions simply 
inherit the formalizations involved in preexisting assessment mechanisms, 
which in the case of human-graded performance reviews, may be far less 
consistent.*” 

Thus, the definition of the target variable and its associated class labels 
will determine what data mining happens to find. While critics of data mining 
have tended to focus on inaccurate classifications (false positives and false 
negatives), ' as much—if not more—danger resides in the definition of the 
class label itself and the subsequent labeling of examples from which rules are 
inferred.” While different choices for the target variable and class labels can 
seem more or less reasonable, valid concerns with discrimination enter at this 
stage because the different choices may have a greater or lesser adverse impact 
on protected classes. For example, as later Parts will explain in detail, hiring 
decisions made on the basis of predicted tenure are much more likely to have a 
disparate impact on certain protected classes than hiring decisions that turn on 
some estimate of worker productivity. If the turnover rate happens to be 
systematically higher among members of certain protected classes, hiring 
decisions based on predicted length of employment will result in fewer job 
opportunities for members of these groups, even if they would have performed 
as well as or better than the other applicants the company chooses to hire. 


B. Training Data 


As described above, data mining learns by example. Accordingly, what a 
model learns depends on the examples to which it has been exposed. The data 
that function as examples are known as “training data”—quite literally, the data 
that train the model to behave in a certain way. The character of the training 
data can have meaningful consequences for the lessons that data mining 
happens to learn. As computer science scholars explain, biased training data 
leads to discriminatory models.” This can mean two rather different things, 


30. Joseph M. Stauffer & M. Ronald Buckley, The Existence and Nature of Racial Bias in 
Supervisory Ratings, 90 J. APPLIED PSYCHOL. 586, 588-89 (2005) (showing evidence of racial bias in 
performance evaluations). Nevertheless, devising new target variables can have the salutary effect of 
forcing decision makers to think much more concretely about the outcomes that justifiably determine 
whether someone is a “good” employee. The explicit enumeration demanded of data mining thus also 
presents an opportunity to make decision making more consistent, more accountable, and fairer 
overall. This, however, requires conscious effort and careful thinking, and is not a natural consequence 
of adopting data mining. 
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N2KR]; Oscar H. Gandy Jr., Engaging Rational Discrimination: Exploring Reasons for Placing 
Regulatory Constraints on Decision Support Systems, 12 ETHICS & INFO. TECH. 29, 39-40 (2010); 
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DISCRIMINATION AND PRIVACY IN THE INFORMATION SOCIETY 3, 20 (Bart Custers et al. eds., 2013). 
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though: (1) if data mining treats cases in which prejudice has played some role 
as valid examples to learn from, that rule may simply reproduce the prejudice 
involved in these earlier cases; or (2) if data mining draws inferences from a 
biased sample of the population, any decision that rests on these inferences 
may systematically disadvantage those who are under- or overrepresented in 
the dataset. Both can affect the training data in ways that lead to discrimination, 
but the mechanisms—improper labeling of examples and biased data 
collections—are sufficiently distinct that they warrant separate treatment. 


1. Labeling Examples 


Labeling examples is the process by which the training data is manually 
assigned class labels. In cases of fraud or spam, the data miners draw from 
examples that come prelabeled: when individual customers report fraudulent 
charges or mark a message as spam, they are actually labeling transactions and 
email for the providers of credit and webmail. Likewise, an employer using 
grades previously given at performance reviews is also using prelabeled 
examples. 

In certain cases, however, there may not be any labeled data and data 
miners may have to figure out a way to label examples themselves. This can be 
a laborious process, and it is frequently fraught with peril. Often the best 
labels for different classifications will be open to debate. On which side of the 
creditworthy line does someone who has missed four credit card payments fall, 
for example?” The answer is not obvious. Even where the class labels are 
uncontested or uncontroversial, they may present a problem because analysts 
will often face difficult choices in deciding which of the available labels best 
applies to a particular example. Certain cases may present some, but not all, 
criteria for inclusion in a particular class.*° The situation might also work in 
reverse, where the class labels are insufficiently precise to capture meaningful 
differences between cases. Such imperfect matches will demand that data 
miners exercise judgment. 

The unavoidably subjective labeling of examples will skew the resulting 
findings such that any decisions taken on the basis of those findings will 
characterize all future cases along the same lines. This is true even if such 


34. Hand, supra note 26, at 10-11. 

35. Id. at 10 (“The classical supervised classification paradigm also takes as fundamental the 
fact that the classes are well defined. That is, that there is some fixed clear external criterion, which is 
used to produce the class labels. In many situations, however, this is not the case. In particular, when 
the classes are defined by thresholding a continuous variable, there is always the possibility that the 
defining threshold might be changed. Once again, this situation arises in consumer credit, where it is 
common to define a customer as ‘defaulting’ if they fall three months in arrears with repayments. This 
definition, however, is not a qualitative one (contrast has a tumor/does not have a tumor) but is very 
much a quantitative one. It is entirely reasonable that alternative definitions (e.g., four months in 
arrears) might be more useful if economic conditions were to change.”). 
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characterizations would seem plainly erroneous to analysts who looked more 
closely at the individual cases. For all their potential problems, though, the 
labels applied to the training data must serve as ground truth.*’ Thus, decisions 
based on discoveries that rest on haphazardly labeled data or data labeled in a 
systematically, though unintentionally, biased manner will seem valid 
according to the customary validation methods employed by data miners. So 
long as prior decisions affected by some form of prejudice serve as examples of 
correctly rendered determinations, data mining will necessarily infer rules that 
exhibit the same prejudice. 

Consider a real-world example from a different context as to how biased 
data labeling can skew results. St. George’s Hospital, in the United Kingdom, 
developed a computer program to help sort medical school applicants based on 
its previous admissions decisions.*® Those admissions decisions, it turns out, 
had systematically disfavored racial minorities and women with credentials 
otherwise equal to other applicants’.°° In drawing rules from biased prior 
decisions, St. George’s Hospital unknowingly devised an automated process 
that possessed these very same prejudices. As editors at the British Medical 
Journal noted at the time, “[T]he program was not introducing new bias but 
merely reflecting that already in the system.”*° Were an employer to undertake 
a similar plan to automate its hiring decisions by inferring a rule from past 
decisions swayed by prejudice, the employer would likewise arrive at a 
decision procedure that simply reproduces the prejudice of prior decision 
makers. Indeed, automating the process in this way would turn the conscious 
prejudice or implicit bias of individuals involved in previous decision making 
into a formalized rule that would systematically alter the prospects of all future 
applicants. For example, the computer may learn to discriminate against certain 
female or black applicants if trained on prior hiring decisions in which an 
employer has consistently rejected jobseekers with degrees from women’s or 
historically black colleges. 

Not only can data mining inherit prior prejudice through the mislabeling 
of examples, it can also reflect current prejudice through the ongoing behavior 
of users taken as inputs to data mining. This is what Professor Latanya 
Sweeney discovered in a study that found that Google queries for black- 
sounding names were more likely to return contextual (i.e., key-word triggered) 


37. Id. at 12. Even when evaluating a model, the kinds of subtle mischaracterizations that 
happen during training will be impossible to detect because most “evaluation data” is just a small 
subset of the training data that has been withheld during the learning process. Any problems with the 
training data will be present in the evaluation data. 

38. Stella Lowry & Gordon Macpherson, A Blot on the Profession, 296 BRIT. MED. J. 657, 
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39. Id. at 657. 
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advertisements for arrest records than those for white-sounding names.*! 
Sweeney confirmed that the companies paying for these advertisements had not 
set out to focus on black-sounding names; rather, the fact that black-sounding 
names were more likely to trigger such advertisements seemed to be an artifact 
of the algorithmic process that Google employs to determine which 
advertisements to display alongside certain queries.” Although it is not fully 
known how Google computes the so-called “quality score” according to which 
it ranks advertisers’ bids, one important factor is the predicted likelihood, based 
on historical trends, that users will click on an advertisement.” As Sweeney 
points out, the process “learns over time which [advertisement] text gets the 
most clicks from viewers [of the advertisement]” and promotes that 
advertisement in its rankings accordingly.“ Sweeney posits that this aspect of 
the process could result in the differential delivery of advertisements that 
reflect the kinds of prejudice held by those exposed to the advertisements." In 
attempting to cater to users’ preferences, Google will unintentionally reproduce 
the existing prejudices that inform users’ choices. 

A similar situation could conceivably arise on websites that recommend 
potential employees to employers, as LinkedIn does through its Talent Match 
feature.*° If LinkedIn determines which candidates to recommend based on the 
demonstrated interest of employers in certain types of candidates, Talent Match 
will offer recommendations that reflect whatever biases employers happen to 
exhibit. In particular, if LinkedIn’s algorithm observes that employers disfavor 
certain candidates who are members of a protected class, Talent Match may 
decrease the rate at which it recommends these candidates to employers. The 
recommendation engine would learn to cater to the prejudicial preferences of 
employers. 

There is an old adage in computer science: “garbage in, garbage out.” 
Because data mining relies on training data as ground truth, when those inputs 
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are themselves skewed by bias or inattention, the resulting system will produce 
results that are at best unreliable and at worst discriminatory. 


2. Data Collection 


Decisions that depend on conclusions drawn from incorrect, partial, or 
nonrepresentative data may discriminate against protected classes. The 
individual records that a company maintains about a person might have serious 
mistakes,’” the records of the entire protected class of which this person is a 
member might also have similar mistakes at a higher rate than other groups, 
and the entire set of records may fail to reflect members of protected classes in 
accurate proportion to others. In other words, the quality and 
representativeness of records might vary in ways that correlate with class 
membership (e.g., institutions might maintain systematically less accurate, 
precise, timely, and complete records for certain classes of people). Even a 
dataset with individual records of consistently high quality can suffer from 
statistical biases that fail to represent different groups in accurate proportions. 
Much attention has focused on the harms that might befall individuals whose 
records in various commercial databases are error ridden.” Far less 
consideration, however, has been paid to the systematic disadvantage that 
members of protected classes may suffer from being miscounted and, as a 
result, misrepresented in the evidence base. 

Recent scholarship has begun to stress this point. Jonas Lerman, for 
example, worries about “the nonrandom, systemic omission of people who live 
on big data’s margins, whether due to poverty, geography, or lifestyle, and 
whose lives are less ‘datafied’ than the general population’s.”*° Professor Kate 
Crawford has likewise warned that “[b]ecause not all data is created or even 
collected equally, there are ‘signal problems’ in big-data sets—dark zones or 
shadows where some citizens and communities are overlooked or 


47. Data quality is a topic of lively practical and philosophical debate. See, e.g., Luciano 
Floridi, Information Quality, 26 PHIL. & TECH. 1 (2013); Richard Y. Wang & Diane M. Strong, 
Beyond Accuracy: What Data Quality Means to Data Consumers, 12 J. MGMT. INFO. Sys. 5 (1996). 
The components of data quality have been thought to include accuracy, precision, completeness, 
consistency, validity, and timeliness, though this catalog of features is far from settled. See generally 
LARRY P. ENGLISH, INFORMATION QUALITY APPLIED (2009). 
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underrepresented.”°”' Errors of this sort may befall historically disadvantaged 
groups at higher rates because they are less involved in the formal economy 
and its data-generating activities, have unequal access to and relatively less 
fluency in the technology necessary to engage online, or are less profitable 
customers or important constituents and therefore less interesting as targets of 
observation.” Not only will the quality of individual records of members of 
these groups be poorer as a consequence, but these groups as a whole will also 
be less well represented in datasets, skewing conclusions that may be drawn 
from an analysis of the data. 

As an illustrative example, Crawford points to Street Bump, an 
application for Boston residents that takes advantage of accelerometers built 
into smart phones to detect when drivers ride over potholes.” While Crawford 
praises the cleverness and cost-effectiveness of this passive approach to 
reporting road problems, she rightly warns that whatever information the city 
receives from Street Bump will be biased by the uneven distribution of 
smartphones across populations in different parts of the city. In particular, 
systematic differences in smartphone ownership will very likely result in the 
underreporting of road problems in the poorer communities where protected 
groups disproportionately congregate.” If the city were to rely on this data to 
determine where it should direct its resources, it would only further underserve 
these communities. Indeed, the city would discriminate against those who lack 
the capability to report problems as effectively as wealthier residents with 
smartphones.” 

A similar dynamic could easily apply in an employment context if 
members of protected classes are unable to report their interest in and 
qualification for jobs listed online as easily or effectively as others due to 
systematic differences in Internet access. The EEOC has established a program 
called “Eradicating Racism & Colorism from Employment” (E-RACE) that 
aims, at least in part, to prevent this sort of discrimination from occurring due 


51. Kate Crawford, Think Again: Big Data, FOREIGN POL’y (May 10, 2013), 
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52. See id.; Lerman, supra note 50, at 57. 

53. Crawford, supra note 51 (explaining that a sudden movement suggesting a broken road 
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55. See id. 
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See, eg., Larry M. Bartels, Economic Inequality and Political Representation, in THE 
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to an employer’s desire for high-tech hiring, such as video résumés.” E-RRACE 
not only attempts to lower the barriers that would disproportionately burden 
applicants who belong to a protected class, but also ensures that employers do 
not develop an inaccurate impression of the incidence of qualified and 
interested candidates from these communities. If employers were to rely on 
tallies of high-tech candidates to direct their recruiting efforts, for example, any 
count affected by a reporting bias could have adverse consequences for specific 
populations systematically underrepresented in the dataset. Employers would 
deny equal attention to those who reside in areas incorrectly pegged as having a 
relatively lower concentration of qualified candidates. 

Additional and even more severe risks may reside in the systematic 
omission of members of protected classes from such datasets. The Street Bump 
and Internet job application examples only discuss decisions that depend on 
raw tallies, rather than datasets from which decision makers want to draw 
generalizations and generate predictions. But data mining is especially sensitive 
to statistical bias because data mining helps to discover patterns that 
organizations tend to treat as generalizable findings even though the analyzed 
data only includes a partial sample from a circumscribed period. To ensure that 
data mining reveals patterns that hold true for more than the particular sample 
under analysis, the sample must be proportionally representative of the entire 
population, even though the sample, by definition, does not include every 
case.” 

If a sample includes a disproportionate representation of a particular class 
(more or less than its actual incidence in the overall population), the results of 
an analysis of that sample may skew in favor of or against the over- or 
underrepresented class. While the representativeness of the data is often simply 
assumed, this assumption is rarely justified and is “perhaps more often 
incorrect than correct.”® Data gathered for routine business purposes tend to 
lack the rigor of social scientific data collection.” As Lerman points out, 
“Businesses may ignore or undervalue the preferences and behaviors of 
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consumers who do not shop in ways that big data tools can easily capture, 
aggregate, and analyze.”°! 

In the employment context, even where a company performs an analysis 
of the data from its entire population of employees—avoiding the apparent 
problem of even having to select a sample—the organization must assume that 
its future applicant pool will have the same degree of variance as its current 
employee base. An organization’s tendency, however, to perform such analyses 
in order to change the composition of their employee base should put the 
validity of this assumption into immediate doubt. The potential effect of this 
assumption is the future mistreatment of individuals predicted to behave in 
accordance with the skewed findings derived from the biased sample. Worse, 
these results may lead to decision procedures that limit the future contact an 
organization will have with specific groups, skewing still further the sample 
upon which subsequent analyses will be performed.” Limiting contact with 
specific populations on the basis of unsound generalizations may deny 
members of these populations the opportunity to prove that they buck the 
apparent trend. 

Overrepresentation in a dataset can also lead to disproportionately high 
adverse outcomes for members of protected classes. Consider an example from 
the workplace: managers may devote disproportionate attention to monitoring 
the activities of employees who belong to a protected class and consequently 
observe mistakes and transgressions at systematically higher rates than others, 
in part because these managers fail to subject others who behave similarly to 
the same degree of scrutiny. Not only does this provide managers with 
justification for their prejudicial suspicions, but it also generates evidence that 
overstates the relative incidence of offenses by members of these groups. 
Where subsequent managers who hold no such prejudicial suspicions cannot 
observe everyone equally, they may rely on this evidence to make predictions 
about where to focus their attention in the future and thus further increase the 
disproportionate scrutiny that they place on protected classes. 

The efficacy of data mining is fundamentally dependent on the quality of 
the data from which it attempts to draw useful lessons. If these data capture the 
prejudicial or biased behavior of prior decision makers, data mining will learn 
from the bad example that these decisions set. If the data fail to serve as a good 
sample of a protected group, data mining will draw faulty lessons that could 
serve as a discriminatory basis for future decision making. 
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C. Feature Selection 


Through a process called “feature selection,” organizations—and the data 
miners that work for them—make choices about what attributes they observe 
and subsequently fold into their analyses.” These decisions can also have 
serious implications for the treatment of protected classes if those factors that 
better account for pertinent statistical variation among members of a protected 
class are not well represented in the set of selected features.“ Members of 
protected classes may find that they are subject to systematically less accurate 
classifications or predictions because the details necessary to achieve equally 
accurate determinations reside at a level of granularity and coverage that the 
selected features fail to achieve. 

This problem arises because data are necessarily reductive representations 
of an infinitely more specific real-world object or phenomenon.” These 
representations may fail to capture enough detail to allow for the discovery of 
crucial points of contrast. Increasing the resolution and range of the analysis 
may still fail to capture the mechanisms that account for different outcomes 
because such mechanisms may not lend themselves to exhaustive or effective 
representation in the data, if such representations even exist. As Professors 
Toon Calders and Indrė Zliobaité explain, “[I]t is often impossible to collect all 
the attributes of a subject or take all the environmental factors into account 
with a model.”®° While these limitations lend credence to the argument that a 
dataset can never fully encompass the full complexity of the individuals it 
seeks to represent, they do not reveal the inherent inadequacy of representation 
as such. 

At issue, really, are the coarseness and comprehensiveness of the criteria 
that permit statistical discrimination and the uneven rates at which different 
groups happen to be subject to erroneous determinations. Crucially, these 
erroneous and potentially adverse outcomes are artifacts of statistical reasoning 
rather than prejudice on the part of decision makers or bias in the composition 
of the dataset. As Professor Frederick Schauer explains, decision makers that 
rely on statistically sound but nonuniversal generalizations “are being 
simultaneously rational and unfair” because certain individuals are “actuarially 
saddled” by statistically sound inferences that are nevertheless inaccurate.“ 
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Obtaining information that is sufficiently rich to permit precise distinctions can 
be expensive. Even marginal improvements in accuracy may come at 
significant practical costs and may justify a less granular and encompassing 
analysis. 

To take an obvious example from the employment context, hiring 
decisions that consider academic credentials tend to assign enormous weight to 
the reputation of the college or university from which an applicant has 
graduated, even though such reputations may communicate very little about the 
applicant’s job-related skills and competencies.” If equally competent 
members of protected classes happen to graduate from these colleges or 
universities at disproportionately low rates, decisions that turn on the 
credentials conferred by these schools, rather than some set of more specific 
qualities that more accurately sort individuals, will incorrectly and 
systematically discount these individuals. Even if employers have a rational 
incentive to look beyond credentials and focus on criteria that allow for more 
precise and more accurate determinations, they may continue to favor 
credentials because they communicate pertinent information at no cost to the 
employer.” 

Similar dynamics seem to account for the practice known as “redlining,””’ 
in which financial institutions employ especially general criteria to draw 
distinctions between subpopulations (i.e., the neighborhood in which 
individuals happen to reside), despite the fact that such distinctions fail to 
capture significant variation within each subpopulation that would result in a 
different assessment for certain members of these groups. While redlining in 
America is well known to have had its basis in racial animus and prejudice,” 
decision makers operating in this manner may attempt to justify their behavior 
by pointing to the cost efficiency of relying on easily accessible information. In 
other words, decision makers can argue that they are willing to tolerate higher 
rates of erroneous determinations for certain groups because the benefits 
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derived from more granular data—and thus better accuracy—do not justify the 
costs. Of course, it may be no coincidence that such cost-benefit analyses seem 
to justify treating groups composed disproportionately of members of protected 
classes to systematically less accurate determinations.” Redlining is illegal 
because it can systematically discount entire areas composed primarily of 
members of a protected class, despite the presence of some qualified 
candidates.”* 

Cases of so-called rational racism are really just a special instance of this 
more general phenomenon—one in which race happens to be taken into 
consideration explicitly. In such cases, decision makers take membership in a 
protected class into account, even if they hold no prejudicial views, because 
such membership seems to communicate relevant information that would be 
difficult or impossible to obtain otherwise. Accordingly, the persistence of 
distasteful forms of discrimination may be the result of a lack of information, 
rather than a continued taste for discrimination.” Professor Lior Strahilevitz 
has argued, for instance, that when employers lack access to criminal records, 
they may consider race in assessing an applicant’s likelihood of having a 
criminal record because there are statistical differences in the rates at which 
members of different racial groups have been convicted of crimes.” In other 
words, employers fall back on more immediately available and coarse features 
when they cannot access more specific or verified information.” Of course, as 
Strahilevitz points out, race is a highly imperfect basis upon which to predict an 
individual’s criminal record, despite whatever differences may exist in the rates 
at which members of different racial groups have been convicted of crimes, 
because it is too coarse as an indicator.” 
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75. See generally Andrea Romei & Salvatore Ruggieri, Discrimination Data Analysis: A 
Multi-Disciplinary Bibliography, in DISCRIMINATION AND PRIVACY IN THE INFORMATION SOCIETY, 
supra note 33, at 109, 120. 

76. Lior Jacob Strahilevitz, Privacy Versus Antidiscrimination, 75 U. CHI. L. REV. 363, 364 
(2008). 

77. Id. This argument assumes that criminal records are relevant to employment, which is 
often not true. See infra text accompanying note 175. 

78.  Strahilevitz, supra note 76, at 364; see also infra Part II.A. The law holds that decision 
makers should refrain from considering membership in a protected class even if statistical evidence 
seems to support certain inferences on that basis. The prohibition does not depend on whether decision 
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D. Proxies 


Cases of decision making that do not artificially introduce discriminatory 
effects into the data mining process may nevertheless result in systematically 
less favorable determinations for members of protected classes. This is possible 
when the criteria that are genuinely relevant in making rational and well- 
informed decisions also happen to serve as reliable proxies for class 
membership. In other words, the very same criteria that correctly sort 
individuals according to their predicted likelihood of excelling at a job—as 
formalized in some fashion—may also sort individuals according to class 
membership. 

In certain cases, there may be an obvious reason for this. Just as “mining 
from historical data may . . . discover traditional prejudices that are endemic in 
reality (i.e., taste-based discrimination),” so, too, may data mining “discover 
patterns of lower performances, skills or capacities of protected-by-law 
groups.” These discoveries not only reveal the simple fact of inequality, but 
they also reveal that these are inequalities in which members of protected 
classes are frequently in the relatively less favorable position. This has rather 
obvious implications: if features held at a lower rate by members of protected 
groups nevertheless possess relevance in rendering legitimate decisions, such 
decisions will necessarily result in systematically less favorable determinations 
for these individuals. For example, by conferring greater attention and 
opportunities to employees that they predict will prove most competent at some 
task, employers may find that they subject members of protected groups to 
consistently disadvantageous treatment because the criteria that determine the 
attractiveness of employees happen to be held at systematically lower rates by 
members of these groups.* 

Decision makers do not necessarily intend this disparate impact because 
they hold prejudicial beliefs; rather, their reasonable priorities as profit seekers 
unintentionally recapitulate the inequality that happens to exist in society. 
Furthermore, this may occur even if proscribed criteria have been removed 
from the dataset, the data are free from latent prejudice or bias, the features are 
especially granular and diverse, and the only goal is to maximize classificatory 
or predictive accuracy. The problem stems from what researchers call 
“redundant encodings,” cases in which membership in a protected class 
happens to be encoded in other data.*' This occurs when a particular piece of 
data or certain values for that piece of data are highly correlated with 


makers can gain (easy or cheap) access to alternative criteria that hold greater predictive value. See 
Grutter v. Bollinger, 539 U.S. 306, 326 (2003). 

79. Romei & Ruggieri, supra note 75, at 121. 

80. Faisal Kamiran, Toon Calders & Mykola Pechenizkiy, Techniques for Discrimination- 
Free Predictive Models, in DISCRIMINATION AND PRIVACY IN THE INFORMATION SOCIETY, supra 
note 33, at 223-24. 

81. Cynthia Dwork et al., Fairness Through Awareness, 3 PROC. INNOVATIONS 
THEORETICAL COMPUTER SCI. CONF. 214 app. at 226 (2012) (“Catalog of Evils”). 
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membership in specific protected classes. Data’s significant statistical 
relevance to the decision at hand helps explain why data mining can result in 
seemingly discriminatory models even when its only objective is to ensure the 
greatest possible accuracy for its determinations. If there is a disparate 
distribution of an attribute, a more precise form of data mining will be more 
likely to capture that distribution. Better data and more features will simply 
come closer to exposing the exact extent of inequality. 


E. Masking 


Data mining could also breathe new life into traditional forms of 
intentional discrimination because decision makers with prejudicial views can 
mask their intentions by exploiting each of the mechanisms enumerated above. 
Stated simply, any form of discrimination that happens unintentionally can also 
be orchestrated intentionally. For instance, decision makers could knowingly 
and purposefully bias the collection of data to ensure that mining suggests rules 
that are less favorable to members of protected classes.*” They could likewise 
attempt to preserve the known effects of prejudice in prior decision making by 
insisting that such decisions constitute a reliable and impartial set of examples 
from which to induce a decision-making rule. And decision makers could 
intentionally rely on features that only permit coarse-grained distinction 
making—distinctions that result in avoidably higher rates of erroneous 
determinations for members of a protected class. In denying themselves finer- 
grained detail, decision makers would be able to justify writing off entire 
groups composed disproportionately of members of protected classes. A form 
of digital redlining, this decision masks efforts to engage in intentional 
discrimination by abstracting to a level of analysis that fails to capture lower 
level variations. As a result, certain members of protected classes might not be 
seen as attractive candidates. Here, prejudice rather than some legitimate 
business reason (such as cost) motivates decision makers to intentionally 
restrict the particularity of their decision making to a level that can only paint 
in avoidably broad strokes. This condemns entire groups, composed 
disproportionately of members of protected classes, to systematically less 
favorable treatment. 

Because data mining holds the potential to infer otherwise unseen 
attributes, including those traditionally deemed sensitive,*’ it can indirectly 
determine individuals’ membership in protected classes and unduly discount, 
penalize, or exclude such people accordingly. In other words, data mining 
could grant decision makers the ability to distinguish and disadvantage 
members of protected classes even if those decision makers do not have access 
to explicit information about individuals’ class membership. Data mining could 


82. See id. (discussing the “[s]elf-fulfilling prophecy”). 
83. See Solon Barocas, Leaps and Bounds: Toward a Normative Theory of Inferential Privacy 
9 (Nov. 11, 2015) (in-progress and unpublished manuscript) (on file with authors). 
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instead help to pinpoint reliable proxies for such membership and thus place 
institutions in the position to automatically sort individuals into their respective 
class without ever having to learn these facts directly.** The most immediate 
implication is that institutions could employ data mining to circumvent the 
barriers, both practical and legal, that have helped to withhold individuals’ 
protected class membership from consideration. 

Additionally, data mining could provide cover for intentional 
discrimination of this sort because the process conceals the fact that decision 
makers determined and considered the individual’s class membership. The 
worry, then, is not simply that data mining introduces novel ways for decision 
makers to satisfy their taste for illegal discrimination; rather, the worry is that it 
may mask actual cases of such discrimination.*° Although scholars, policy 
makers, and lawyers have long been aware of the dangers of masking,” data 
mining significantly enhances the ability to conceal acts of intentional 
discrimination by finding ever more remote and complex proxies for proscribed 
criteria.*” 

Intentional discrimination and its masking have so far garnered 
disproportionate attention in discussions of data mining, often to the 
exclusion of issues arising from the many forms of unintentional discrimination 
described above. While data mining certainly introduces novel ways to 
discriminate intentionally and to conceal those intentions, most cases of 
employment discrimination are already sufficiently difficult to prove; 
employers motivated by conscious prejudice would have little to gain by 
pursuing these complex and costly mechanisms to further mask their 
intentions.* When it comes to data mining, unintentional discrimination is the 
more pressing concern because it is likely to be far more common and easier to 
overlook. 


84. Id. at 9-13. 

85. Data miners who wish to discriminate can do so using relevant or irrelevant criteria. Either 
way the intent would make the action “masking.” If an employer masked using highly relevant data, 
litigation arising from it likely would be tried under a “mixed-motive” framework, which asks whether 
the same action would have been taken without the intent to discriminate. See infra Part ILA. 

86. See, e.g., Custers, supra note 33, at 9-10. 

87. See Barocas, supra note 83. 

88. See, e.g., Alistair Croll, Big Data Is Our Generation’s Civil Rights Issue, and We Don’t 
Know It, SOLVE FOR INTERESTING (July 31, 2012, 12:40 PM), http://solveforinteresting.com/big-data- 
is-our-generations-civil-rights-issue-and-we-dont-know-it [https://perma.cc/BS8S-6T7S]. This post 
generated significant online chatter immediately upon publication and has become one of the 
canonical texts in the current debate. It has also prompted a number of responses from scholars. See, 
e.g., Anders Sandberg, Asking the Right Questions: Big Data and Civil Rights, PRAC. ETHICS (Aug. 
16, 2012), http://blog.practicalethics.ox.ac.uk/2012/08/asking-the-right-questions-big-data-and-civil- 
rights [https://perma.cc/NC36-NBZN]. 

89. See Linda Hamilton Krieger, The Content of Our Categories: A Cognitive Bias Approach 
to Discrimination and Equal Employment Opportunity, 47 STAN. L. REV. 1161, 1177 (1995). 
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I. 
TITLE VII LIABILITY FOR DISCRIMINATORY DATA MINING 


Current antidiscrimination law is not well equipped to address the cases of 
discrimination stemming from the problems described in Part I. This Part 
considers how Title VII might apply to these cases. Other antidiscrimination 
laws, such as the Americans with Disabilities Act, will exhibit differences in 
specific operation, but the main thrust of antidiscrimination law is fairly 
consistent across regimes, and Title VII serves as an illustrative example.” 

An employer sued under Title VII may be found liable for employment 
discrimination under one of two theories of liability: disparate treatment and 
disparate impact.”! Disparate treatment comprises two different strains of 
discrimination: (1) formal disparate treatment of similarly situated people and 
(2) intent to discriminate.” Disparate impact refers to policies or practices that 
are facially neutral but have a disproportionately adverse impact on protected 
classes.” Disparate impact is not concerned with the intent or motive for a 
policy; where it applies, the doctrine first asks whether there is a disparate 
impact on members of a protected class, then whether there is some business 
justification for that impact, and finally, whether there were less discriminatory 
means of achieving the same result.” 

Liability under Title VII for discriminatory data mining will depend on 
the particular mechanism by which the inequitable outcomes are generated. 
This Part explores the disparate treatment and disparate impact doctrines and 
analyzes which mechanisms could generate liability under each theory. 


A. Disparate Treatment 


Disparate treatment recognizes liability for both explicit formal 
classification and intentional discrimination.” Formal discrimination, in which 
membership in a protected class is used as an input to the model, corresponds 
to an employer classifying employees or potential hires according to 
membership in a protected class and differentiating them on that basis. Formal 


90. The biggest difference between the Americans with Disabilities Act and Title VII is the 
requirement that an employer make “reasonable accommodations” for disabilities. 42 U.S.C. 
§ 12112(b)(5) (2012). But some scholars have argued that even this difference is illusory and that 
accommodations law is functionally similar to Title VII, though worded differently. See Samuel R. 
Bagenstos, “Rational Discrimination, ” Accommodation, and the Politics of (Disability) Civil Rights, 
89 VA. L. REV. 825, 833 & n.15 (2003) (comparing accommodations law to disparate treatment); 
Christine Jolls, Antidiscrimination and Accommodation, 115 HARV. L. REV. 642, 652 (2001) 
(comparing accommodations law to disparate impact). 

91. See 42 U.S.C. § 2000e; Ricci v. DeStefano, 557 U.S. 557, 577 (2009). 

92. Richard A. Primus, The Future of Disparate Impact, 108 MICH. L. REV. 1341, 1351 n.56 
(2010) (explaining that, for historical reasons, disparate treatment became essentially “not-disparate- 
impact” and now we rarely notice the two different embedded theories). 

93. See Griggs v. Duke Power Co., 401 U.S. 424, 430 (1971). 

94. 42 U.S.C. § 2000e-2(k). 

95. Id. § 2000e-2(a), (k); see Primus, supra note 92, at 1350-51 n.56. 
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discrimination covers both the straightforward denial of opportunities based on 
protected class membership and the use of rational racism.”° In traditional 
contexts, rational racism is considered rational because there are cases in which 
its users believe it is an accurate, if coarse-grained, proxy—or at least the best 
available one in a given situation.” In the world of data mining, though, that 
need not be the case. Even if membership in a protected class were specified as 
an input, the eventual model that emerges could see it as the least significant 
feature. In that case, there would be no discriminatory effect, but there would 
be a disparate treatment violation, because considering membership in a 
protected class as a potential proxy is a legal classificatory harm in itself.”® 
Formal liability does not correspond to any particular discrimination 
mechanism within data mining; it can occur equally well in any of them. 
Because classification itself can be a legal harm, irrespective of the effect,” the 
same should be true of using protected class as an input to a system for which 
the entire purpose is to build a classificatory model.'* The irony is that the use 
of protected class as an input is usually irrelevant to the outcome in terms of 
discriminatory effect, at least given a large enough number of input features. 
The target variable will, in reality, be correlated to the membership in a 
protected class somewhere between 0 percent and 100 percent. If the trait is 
perfectly uncorrelated, including membership in the protected class as an input 
will not change the output, and there will be no discriminatory effect." On the 
other end of the spectrum, where membership in the protected class is perfectly 
predictive of the target variable, the fact will be redundantly encoded in the 
other data. The only way using membership in the protected class as an explicit 
feature will change the outcome is if the information is otherwise not rich 
enough to detect such membership. Membership in the protected class will 
prove relevant to the exact extent it is already redundantly encoded. Given a 
rich enough set of features, the chance that such membership is redundantly 
encoded approaches certainty. Thus, a data mining model with a large number 
of variables will determine the extent to which membership in a protected class 
is relevant to the sought-after trait whether or not that information is an input. 
Formal discrimination therefore should have no bearing whatsoever on the 


96. Michelle R. Gomez, The Next Generation of Disparate Treatment: A Merger of Law and 
Social Science, 32 REV. LITIG. 553, 562 (2013). 

97. Strahilevitz, supra note 76, at 365-67. 

98. Richard A. Primus, Equal Protection and Disparate Impact: Round Three, 117 HARV. L. 
REV. 494, 504 (2003). 

99. See Jed Rubenfeld, Affirmative Action, 107 YALE L.J. 427, 433 (1997) (discussing 
“Tc]lassificationism’); Primus, supra note 98, at 504, 567-68 (discussing expressive harms). 

100. Membership in a protected class is still a permissible input to a holistic determination 
when the focus is diversity, but where classification is the goal, such as here, it is not. See Grutter v. 
Bollinger, 539 U.S. 306, 325 (2003) (noting that “diversity is a compelling state interest” that can 
survive strict scrutiny). 

101. That is, not counting any expressive harm that might come from classification by protected 
class. 
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outcome of the model. Additionally, by analyzing the data, an employer could 
probabilistically determine an employee’s membership in that same protected 
class, if the employer did indeed want to know. 

To analyze intentional discrimination other than mere formal 
discrimination, a brief description of disparate treatment doctrine is necessary. 
A Title VII disparate treatment case will generally proceed under either the 
McDonnell-Douglas burden-shifting scheme or the Price-Waterhouse “mixed 
motive” regime.’ Under the McDonnell-Douglas framework, the plaintiff 
who has suffered an adverse employment action has the initial responsibility to 
establish a prima facie case of discrimination by demonstrating that a similarly 
situated person who is not a member of a protected class would not have 
suffered the same fate.” This can be shown with circumstantial evidence of 
discriminatory intent, such as disparaging remarks made by the employer or 
procedural irregularities in promotion or hiring; only very rarely will an 
employer openly admit to discriminatory conduct. If the plaintiff successfully 
demonstrates that the adverse action treated protected class members 
differently, then the burden shifts to the defendant-employer to offer a 
legitimate, nondiscriminatory basis for the decision. The defendant need not 
prove the reason is true; his is only a burden of production." Once the 
defendant has offered a nondiscriminatory alternative, the ultimate burden of 
persuasion falls to the plaintiff to demonstrate that the proffered reason is 
pretextual.” 

In the data mining context, liability for masking is clear as a theoretical 
matter, no matter which mechanism for discrimination is employed. The fact 
that it is accomplished algorithmically does not make it less of a disparate 
treatment violation, as the entire idea of masking is pretextual. In fact, in the 
traditional, non—data mining context, the word masking has occasionally been 
used to refer to pretext." Like in any disparate treatment case, however, proof 
will be difficult to come by, something even truer for masking. 


102. McDonnell Douglas Corp. v. Green, 411 U.S. 792 (1973); Price Waterhouse v. Hopkins, 
490 U.S. 228 (1989). 

103. This is similar to the computer science definition of discrimination. Calders & Zliobaité, 
supra note 64, at 49. (“A classifier discriminates with respect to a sensitive attribute, e.g. gender, if for 
two persons which only differ by their gender (and maybe some characteristics irrelevant for the 
classification problem at hand) that classifier predicts different labels.”). 

104. St. Mary’s Honor Ctr. v. Hicks, 509 U.S. 502, 507 (1993). 

105. Id. 

106. See Keyes v. Sec’y of the Navy, 853 F.2d 1016, 1026 (1st Cir. 1988) (explaining that it is 
the plaintiff's burden to show that the proffered reasons for hiring an alternative were “pretexts aimed 
at masking sex or race discrimination”); Custers, supra note 33, at 9-10; Megan Whitehill, Better Safe 
than Subjective: The Problematic Intersection of Prehire Social Networking Checks and Title VI 
Employment Discrimination, 85 TEMP. L. REV. 229, 250 (2012) (referring to “[m]asking [p]retext” in 
the third stage of McDonnell-Douglas framework). 

107. See supra Part I.E. This is a familiar problem to antidiscrimination law, and it is often cited 
as one of the rationales for disparate impact liability in the first place—to “smoke out” intentional 
invidious discrimination. See infra Part II.B. 


2016] BIG DATA’S DISPARATE IMPACT 697 


The McDonnell-Douglas framework operates on a presumption that if the 
rationale that the employer has given is found to be untrue, the employer must 
be hiding his “true” discriminatory motive.” Because the focus of the 
McDonnell-Douglas framework is on pretext and cover-up, it can only address 
conscious, willful discrimination. Under the McDonnell-Douglas 
framework, a court must find either that the employer intended to discriminate 
or did not discriminate at all. "° Thus, unintentional discrimination will not lead 
to liability. 

A Title VII disparate treatment case can also be tried under the mixed- 
motive framework, first recognized in Price Waterhouse v. Hopkins''' and 
most recently modified by Desert Palace, Inc. v. Costa.''” In the mixed-motive 
framework, a plaintiff need not demonstrate that the employer’s 
nondiscriminatory rationale was pretextual, but merely that discrimination was 
a “motivating factor” in the adverse employment action.''? As a practical 
matter, this means that the plaintiff must show that the same action would not 
have been taken absent the discriminatory motive.''* As several commentators 


108. McDonnell Douglas Corp. v. Green, 411 U.S. 792, 805 (1973) (The plaintiff “must be 
given a full and fair opportunity to demonstrate by competent evidence that the presumptively valid 
reasons for his rejection were in fact a coverup for a racially discriminatory decision”). While, as a 
theoretical matter, the plaintiff must prove that the employer’s reason was a pretext for discrimination 
specifically, the Supreme Court has held that a jury can reasonably find that the fact that an employer 
had only a pretextual reason to fall back on is itself circumstantial evidence of discrimination. Hicks, 
509 U.S. at 511 (“The factfinder’s disbelief of the reasons put forward by the defendant (particularly if 
disbelief is accompanied by a suspicion of mendacity) may, together with the elements of the prima 
facie case, suffice to show intentional discrimination.”). 

109. See Tristin K. Green, Discrimination in Workplace Dynamics: Toward a Structural 
Account of Disparate Treatment Theory, 38 HARV. C.R.-C.L. L. REV. 91, 114 (2003) (“Presuming that 
individuals know the real reason for their actions, the pretext model of disparate treatment provides 
that an employer can be held to have discriminated when the plaintiff establishes a minimal prima 
facie case and shows that the reason given for the adverse decision is unworthy of credence.”); Susan 
Sturm, Second Generation Employment Discrimination: A Structural Approach, 101 COLUM. L. REV. 
458, 458 (2001); see also Melissa Hart, Subjective Decisionmaking and Unconscious Discrimination, 
56 ALA. L. REV. 741, 749-50 (2005) (critiquing the courts’ requirement of proving employer 
“dishonesty,” but suggesting that, absent this requirement, Title VII could handle unconscious 
discrimination without altering the law). 

110. Krieger, supra note 89, at 1170. 

111. 490 U.S. 228 (1989). 

112. 539 U.S. 90 (2003). 

113. 42 U.S.C. § 2000e-2(m) (2012); Desert Palace, 539 U.S. at 101 (“In order to obtain [a 
mixed-motive jury instruction], a plaintiff need only present sufficient evidence for a reasonable jury 
to conclude, by a preponderance of the evidence, that ‘race, color, religion, sex, or national origin was 
a motivating factor for any employment practice.’”). The efficacy of data mining is fundamentally 
dependent on the quality of the data from which it attempts to draw useful lessons. If these data 
capture the prejudicial or biased behavior of prior decision makers, data mining will learn from the bad 
example that these decisions set. If the data fail to serve as a good sample of a protected group, data 
mining will draw faulty lessons that could serve as a discriminatory basis for future decision making. 

114. Charles A. Sullivan, Disparate Impact: Looking Past the Desert Palace Mirage, 47 WM. & 
Mary L. REV. 911, 914-16, 916 n.20 (2005); see also Krieger, supra note 89, at 1170-72; D. Don 
Welch, Removing Discriminatory Barriers: Basing Disparate Treatment Analysis on Motive Rather 
than Intent, 60 S. CAL. L. REV. 733, 740 (1987). 
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have pointed out, motive and intent are not necessarily synonymous. '! Motive 
can be read more broadly to include unconscious discrimination, including 
anything that influences a person to act, such as emotions or desires.''® 
Nonetheless, courts have conflated the meanings of motive and intent such that 
the phrase “motive or intent” has come to refer only to conscious choices.'!’ 
Thus, while most individual decision making probably belongs in a mixed- 
motive framework, as each decision a person makes comprises a complicated 
mix of motivations, "S the mixed-motive framework will be no better than the 
pretext framework at addressing bias that occurs absent conscious intent.''” 

Except for masking, discriminatory data mining is by stipulation 
unintentional. Unintentional disparate treatment is not a problem that is new to 
data mining. A vast scholarly literature has developed regarding the law’s 
treatment of unconscious, implicit bias." Such treatment can occur when an 
employer has internalized some racial stereotype and applies it or, without 
realizing it, monitors an employee more closely until the employer finds a 
violation.'*! The employee is clearly treated differently, but it is not intentional, 
and the employer is unaware of it. As Professor Samuel Bagenstos 
summarized, at this point, “it may be difficult, if not impossible, for a court to 
go back and reconstruct the numerous biased evaluations and perceptions that 
ultimately resulted in an adverse employment decision.”'”* Within the scholarly 
literature, there is “[s]urprising unanimity” that the law does not adequately 
address unconscious disparate treatment.'” 


115. Krieger, supra note 89, at 1243; Sullivan, supra note 114, at 915. 

116. Krieger, supra note 89, at 1243; Sullivan, supra note 114, at 915 n.18 (quoting Motive, 
OXFORD ENGLISH DICTIONARY (Ist ed. 1933)). 

117. Sullivan, supra note 114, at 914-16, 916 n.20. 

118. Amy L. Wax, Discrimination as Accident, 74 IND. L.J. 1129, 1149 & n.21 (1999); Krieger, 
supra note 89, at 1223. In fact, after the Supreme Court decided Desert Palace, many scholars thought 
that it had effectively overruled the McDonnell-Douglas framework, forcing all disparate treatment 
cases into a mixed-motive framework. See, e.g., Sullivan, supra note 114, at 933-36 (discussing the 
then-emerging scholarly consensus). This has not played out so far, with courts and scholars split on 
the matter. See, e.g., Kendall D. Isaac, Is It “A” or Is It “The”? Deciphering the Motivating-Factor 
Standard in Employment Discrimination and Retaliation Cases, 1 TEX. A&M L. REV. 55, 74 (2013) 
(“McDonnell Douglas has never been overruled and remains widely utilized.”); Barrett S. Moore, 
Shifting the Burden: Genuine Disputes and Employment Discrimination Standards of Proof, 35 U. 
ARK. LITTLE ROCK L. REV. 113, 123-29, 128 n.146 (2012) (noting a circuit split on the issue). 

119. See Krieger, supra note 89, at 1182-83. 

120. See, e.g., Christine Jolls & Cass R. Sunstein, The Law of Implicit Bias, 94 CALIF. L. REV. 
969, 978 n.45 (2006) (collecting sources); Linda Hamilton Krieger & Susan T. Fiske, Behavioral 
Realism in Employment Discrimination Law: Implicit Bias and Disparate Treatment, 94 CALIF. L. 
REV. 997, 1003 n.21 (2006) (collecting sources). 

121. This example can be ported directly to data mining as overrepresentation in data 
collection. See supra Part I.B.2. 

122. Samuel R. Bagenstos, The Structural Turn and the Limits of Antidiscrimination Law, 94 
CALIF. L. REV. 1, 9 (2006). 

123. Sullivan, supra note 114, at 1000. There is, however, no general agreement on whether the 
law should treat such discrimination as disparate treatment or disparate impact. Compare Krieger, 
supra note 89, at 1231 (explaining that because the bias causes employers to treat people differently, it 
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There are a few possible ways to analogize discriminatory data mining to 
unintentional disparate treatment in the traditional context, based on where one 
believes the “treatment” lies. Either the disparate treatment occurs at the 
decision to apply a predictive model that will treat members of a protected 
class differently, or it occurs when the disparate result of the model is used in 
the ultimate hiring decision. In the first scenario, the intent at issue is the 
decision to apply a predictive model with known disproportionate impact on 
protected classes. In the second, the disparate treatment occurs if, after the 
employer sees the disparate result, he proceeds anyway. If the employer 
continues because he liked the discrimination produced in either scenario, then 
intent is clear. If not, then this just devolves into a standard disparate impact 
scenario, with liability based on effect. Under disparate impact theory, deciding 
to follow through on a test with discriminatory effect does not suddenly render 
it disparate treatment.” 

Another option is to imagine the model as the decision maker exhibiting 
implicit bias. That is, because of biases hidden to the predictive model such as 
nonrepresentative data or mislabeled examples, the model reaches a 
discriminatory result. This analogy turns every mechanism except proxy 
discrimination into the equivalent of implicit bias exhibited by individual 
decision makers. The effect of bias is one factor among the many different 
factors that go into the model-driven decision, just like in an individual’s 
adverse employment decision.'”> Would a more expansive definition of motive 
fix this scenario? 

Because the doctrine focuses on human decision makers as discriminators, 
the answer is no. Even if disparate treatment doctrine could capture 
unintentional discrimination, it would only address such discrimination 
stemming from human bias. For example, the person who came up with the 
idea for Street Bump ultimately devised a system that suffers from reporting 
bias,” but it was not because he or she was implicitly employing some racial 
stereotype. Rather, it was simply inattentiveness to problems with the sampling 
frame. This is not to say that his or her own bias had nothing to do with it—the 
person likely owned a smartphone and thus did not think about the people who 
do not—but no one would say that it was even implicit bias against protected 


should be considered a disparate treatment violation), with Sullivan, supra note 114, at 969-71 
(arguing that the purpose of disparate impact is a catch-all provision to address those types of bias that 
disparate treatment cannot reach). This disagreement is important and even more pronounced in the 
case of data mining. See infra Part III. For now, we assume each case can be analyzed separately. 

124. In fact, after Ricci v. DeStefano, 557 U.S. 557 (2009), deciding not to apply such a test 
after noticing the discriminatory effect may give rise to a disparate treatment claim in the other 
direction. 

125. Bagenstos, supra note 122, at 9; Krieger, supra note 89, at 1185-86 (“Not only disparate 
treatment analysis, but the entire normative structure of Title VII’s injunction ‘not to discriminate,’ 
rests on the assumption that decisionmakers possess ‘transparency of mind’—that they are aware of 
the reasons why they are about to make, or have made, a particular employment decision.”). 

126. See supra note 51 and accompanying text. 
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classes that motivated the decision, even under the expansive definition of the 
word “motive.” 7 

The only possible analogy relevant to disparate treatment, then, is to those 
data mining mechanisms of unintentional discrimination that reflect a real 
person’s bias—something like LinkedIn’s Talent Match recommendation 
engine, which relies on potentially prejudiced human assessments of 
employees. '** As a general rule, an employer may not avoid disparate treatment 
liability by encoding third-party preferences as a rationale for a hiring 
decision.” But, once again, to be found liable under current doctrine, the 
employer would likely both have to know that this is the specific failure 
mechanism of the model and choose it based on this fact. 

There is one other interesting question regarding disparate treatment 
doctrine: whether the intent standard includes knowledge. This is not a problem 
that arises often when a human is making a single employment determination. 
Assuming disparate treatment occurs in a given case, it is generally either 
intended or unconscious. What would it mean to have an employer know that 
he was treating an employee differently, but still take the action he had always 
planned to take without intent to treat the employee differently? It seems like 
an impossible line to draw. ° 

With data mining, though, unlike unconscious bias, it is possible to audit 
the resulting model and inform an employer that she will be treating individuals 
differently before she does so. If an employer intends to employ the model, but 
knows it will produce a disparate impact, does she intend to discriminate? This 
is a more realistic parsing of intent and knowledge than in the case of an 
individual, nonsystematic employment decision. Neither pretext nor motive 
exists here, and throughout civil and criminal law, “knowledge” and “intent” 
are considered distinct states of mind, so there would likely be no liability. On 
the other hand, courts may use knowledge of discrimination as evidence to find 
intent.'*! And while the statute’s language only covers intentional 
discrimination,” a broad definition of intent could include knowledge or 


127. Of course, the very presumption of a design’s neutrality is itself a bias that may work 
against certain people. See Langdon Winner, Do Artifacts Have Politics?, 109 DAEDALUS 121, 125 
(1980). But, as this is a second-order effect, we need not address it here. 

128. See Woods, supra note 46. 

129. See 29 C.F.R. § 1604.2(a)(1)(iii) (2015) (stating the EEOC’s position that “the preferences 
of coworkers, the employer, clients or customers” cannot be used to justify disparate treatment); see 
also Fernandez v. Wynn Oil Co., 653 F.2d 1273, 1276-77 (9th Cir. 1981); Diaz v. Pan Am. World 
Airways, Inc., 442 F.2d 385, 389 (5th Cir. 1971). 

130. See Krieger, supra note 89, at 1185 (discussing disparate treatment’s “assumption of 
decisionmaker self-awareness”). 

131. Columbus Bd. of Educ. v. Penick, 443 U.S. 449, 464 (1979) (“[A]ctions having 
foreseeable and anticipated disparate impact are relevant evidence to prove the ultimate fact, forbidden 
purpose.”); Pers. Adm’r of Mass. v. Feeney, 442 U.S. 256, 279 n.25 (1979) (“[W]hen the adverse 
consequences of a law upon an identifiable group are... inevitable... , a strong inference that the 
adverse effects were desired can reasonably be drawn.”). 

132. 42 U.S.C. § 2000e-2(h) (2012). 
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substantial certainty of the result. Because the situation has not come up 
often, the extent of the “intent” required is as yet unknown." 

In sum, aside from rational racism and masking (with some difficulties), 
disparate treatment doctrine does not appear to do much to regulate 
discriminatory data mining. 


B. Disparate Impact 


Where there is no discriminatory intent, disparate impact doctrine should 
be better suited to finding liability for discrimination in data mining. In a 
disparate impact case, a plaintiff must show that a particular facially neutral 
employment practice causes a disparate impact with respect to a protected 
class.'*° If shown, the defendant-employer may “demonstrate that the 
challenged practice is job related for the position in question and consistent 
with business necessity.”'*° If the defendant makes a successful showing to that 
effect, the plaintiff may still win by showing that the employer could have used 
an “alternative employment practice” with less discriminatory results. a 

The statute is unclear as to the required showing for essentially every 
single element of a disparate impact claim. First, it is unclear how much 
disparate impact is needed to make out a prima facie case.” The EEOC, 
charged with enforcing Title VII’s mandate, has created the so-called “four- 
fifths rule” as a presumption of adverse impact: “A selection rate for any race, 
sex, or ethnic group which is less than four-fifths ... of the rate for the group 


133. See Julia Kobick, Note, Discriminatory Intent Reconsidered: Folk Concepts of 
Intentionality and Equal Protection Jurisprudence, 45 HARV. C.R.-C.L. L. REv. 517, 551 (2010) 
(arguing that courts should regularly consider knowledge and foreseeability of disparate impact as an 
intended effect); cf RESTATEMENT (SECOND) OF TORTS § 8A cmt. b (AM. LAW INST. 1965) (“Intent 
is not... limited to consequences which are desired. If the actor knows that the consequences are 
certain, or substantially certain, to result from his act, and still goes ahead, he is treated by the law as if 
he had in fact desired to produce the result.”). 

134. Determining that a model is discriminatory is also like trying and failing to validate a test 
under disparate impact doctrine. See infra Part II.B. If a test fails validation, the employer using it 
would know that he is discriminating if he applies it, but that does not imply that he is subject to 
disparate treatment liability. Nonetheless, validation is part of the business necessity defense, and that 
defense is not available against disparate treatment claims. Thus, the analysis does not necessarily have 
the same result. 42 U.S.C. § 2000e-2(k)(2). One commentator has argued that including knowledge as 
a state of mind leading to disparate treatment liability would effectively collapse disparate impact and 
disparate treatment by conflating intent and effect. Jessie Allen, A Possible Remedy for Unthinking 
Discrimination, 61 BROOK. L. REV. 1299, 1314 (1995). But others still have noted that with respect to 
knowledge, a claim is still about the treatment of an individual, not the incidental disparate impact of a 
neutral policy. See Carin Ann Clauss, Comparable Worth—The Theory, Its Legal Foundation, and the 
Feasibility of Implementation, 20 U. MICH. J.L. REFORM 7, 62 (1986). 

135. 42 U.S.C. § 2000e-2(k)(1)(A). 

136. Id. 

137. Id. 

138. The statute does not define the requirement and Supreme Court has never addressed the 
issue. See, e.g., Sullivan, supra note 114, at 954 & n.153. For a brief discussion of the different 
approaches to establishing disparate impact, see Pamela L. Perry, Two Faces of Disparate Impact 
Discrimination, 59 FORDHAM L. REV. 523, 570-74 (1991). 
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with the highest rate will generally be regarded... as evidence of adverse 
impact.” The Uniform Guidelines on Employment Selection Procedures 
(Guidelines) also state, however, that smaller differences can constitute adverse 
impact and greater differences may not, depending on circumstances. Thus, the 
four-fifths rule is truly just a guideline." For the purposes of this Part, it is 
worthwhile to just assume that the discriminatory effects are prominent enough 
to establish disparate impact as an initial matter. "*' 

The next step in the litigation is the “business necessity” defense. This 
defense is, in a very real sense, the crux of disparate impact analysis, weighing 
Title VII’s competing goals of limiting the effects of discrimination while 
allowing employers discretion to advance important business goals. Griggs v. 
Duke Power Co.” —the decision establishing the business necessity defense 
alongside disparate impact doctrine itself—articulated the defense in several 
different ways: 

A challenged employment practice must be “shown to be related to job 
performance,” have a “manifest relationship to the employment in 
question,” be “demonstrably a reasonable measure of job 
performance,” bear some “relationship to job-performance ability,” 
and/or “must measure the person for the job and not the person in the 
abstract.”!*° 
The Supreme Court was not clear on what, if any, difference existed between 
job-relatedness and business necessity, at one point seeming to use the terms 
interchangeably: “The touchstone is business necessity. If an employment 
practice which operates to exclude Negroes cannot be shown to be related to 
job performance, the practice is prohibited.”'“* The focus of the Court was 
clearly on future job performance, and the term “job-related” has come to mean 
a practice that is predictive of job performance.“ Because the definitions of 
job-relatedness and business necessity have never been clear, courts defer when 
applying the doctrine and finding the appropriate balance.'*° 

Originally, the business necessity defense seemed to apply narrowly. In 
Griggs, Duke Power had instituted new hiring requirements including a high 
school diploma and success on a “general intelligence” test for previously 


139. Uniform Guidelines on Employment Selection Procedures, 29 C.F.R. § 1607.4(D) (2015) 
[hereinafter Guidelines]. 


140. Id. 
141. We will return to this when discussing the need to grapple with substantive fairness. See 
infra Part II.B. 


142. 401 US. 424 (1971). 

143. Linda Lye, Comment, Title VII's Tangled Tale: The Erosion and Confusion of Disparate 
Impact and the Business Necessity Defense, 19 BERKELEY J. EMP. & LAB. L. 315, 321 (1998) 
(footnotes omitted) (quoting Griggs v. Duke Power Co., 401 U.S. 424, 431-36 (1971)). 

144. Griggs, 401 U.S. at 431; see also Lye, supra note 143, at 320. 

145. Lye, supra note 143, at 355 & n.206. 

146. Td. at 319-20, 348-53; Amy L. Wax, Disparate Impact Realism, 53 WM. & MARY L. 
REV. 621, 633-34 (2011). 
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white-only divisions. Duke Power did not institute such requirements in 
divisions where it had previously hired black employees." The Court ruled 
that the new requirements were not a business necessity because “employees 
who have not completed high school or taken the tests have continued to 
perform satisfactorily and make progress in departments for which the high 
school and test criteria are now used.”'“’ Furthermore, the requirements were 
implemented without any study of their future effect." The Court also rejected 
the argument that the requirements would improve the “overall quality of the 
workforce.”'”° 

By 1979, the Court began treating business necessity as a much looser 
standard.'°' In New York City Transit Authority v. Beazer,” the transit 
authority had implemented a rule barring drug users from employment, 
including current users of methadone, otherwise known as recovering heroin 
addicts. In dicta, the Court stated that a “narcotics rule,” which “significantly 
serves” the “legitimate employment goals of safety and efficiency,” was 
“assuredly” job related.'°’ This was the entire analysis of the business necessity 
defense in the case. Moreover, the rationale was acceptable as applied to the 
entire transit authority, even where only 25 percent of the jobs were labeled as 
“safety sensitive.”'** Ten years later, the Court made the business necessity 
doctrine even more defendant-friendly in Wards Cove Packing Co. v. Atonio.’ 
After Wards Cove, the business necessity defense required a court to engage in 
“a reasoned review of the employer’s justification for his use of the challenged 
practice.... [T]here is no requirement that the challenged practice be 
‘essential’ or ‘indispensable’ to the employer’s business for it to pass 
muster . . . .”'°° The Court also reallocated the burden to plaintiffs to prove that 
business necessity was lacking and even referred to the defense as a “business 
justification” rather than a business necessity." The Wards Cove Court went 
so far that Congress directly addressed the decision in the Civil Rights Act of 
1991 (1991 Act), which codified disparate impact and reset the standards to the 
day before Wards Cove was decided.'** 

Because the substantive standards for job-relatedness or business 
necessity were uncertain before Wards Cove, however, the confusion persisted 


147. Griggs, 401 U.S. at 427-28. 

148. Id. at 431-32. 

149. Id. at 432. 

150. Id. at 431. 

151. See Nicole J. DeSario, Reconceptualizing Meritocracy: The Decline of Disparate Impact 
Discrimination Law, 38 HARV. C.R.-C.L. L. REV. 479, 495-96 (2003); Lye, supra note 143, at 328. 

152. 440 U.S. 568 (1979). 

153. Id. at 587 & n.31. 

154. Id. 

155. 490 U.S. 642 (1989). 

156. Id. at 659. 

157. Id. 

158. 42 U.S.C. § 2000e-2(k)(1)(C) (2012). 


704 CALIFORNIA LAW REVIEW [Vol. 104:671 


even after the 1991 Act was passed.’ At the time, both sides—civil rights 
groups and the Bush administration, proponents of a rigorous and more lenient 
business necessity defense respectively—declared victory. "6 

Since then, courts have recognized that business necessity lies somewhere 
in the middle of two extremes.'*' Some courts require that the hiring criteria 
bear a “manifest relationship”’™ to the employment in question or that they be 
“significantly correlated” to job performance.'® The Third Circuit was briefly 
an outlier, holding “that hiring criteria must effectively measure the ‘minimum 
qualifications for successful performance of the job’” in order to meet the strict 
business necessity standard.’ This tougher standard would, as a practical 
matter, ban general aptitude tests with any disparate impact because a particular 
cutoff score cannot be shown to distinguish between those able and completely 
unable to do the work." For example, other unmeasured skills and abilities 
could theoretically compensate for the lower score on an aptitude test, 
rendering a certain minimum score not “necessary” if it does not measure 
minimum qualifications.’ In a subsequent case, however, the Third Circuit 
recognized that Title VII does not require an employer to choose someone “less 
qualified” (as opposed to unqualified) in the name of nondiscrimination and 
noted that aptitude tests can be legitimate hiring tools if they accurately 
measure a person’s qualifications.'*’ The court concluded: 


159. Legislative history was no help either. The sole piece of legislative history is an 
“interpretive memorandum” that specifies that the standards were to revert to before Wards Cove, 
coupled with an explicit instruction in the Act to ignore any other legislative history regarding business 
necessity. Susan S. Grover, The Business Necessity Defense in Disparate Impact Discrimination 
Cases, 30 GA. L. REV. 387, 392-93 (1996). 

160. Andrew C. Spiropoulos, Defining the Business Necessity Defense to the Disparate Impact 
Cause of Action: Finding the Golden Mean, 74 N.C. L. REV. 1479, 1484 (1996). 

161. Though courts generally state the standard to reflect this middle position, the Supreme 
Court’s latest word on disparate impact—in which the Court reaffirmed the doctrine generally and 
held that it applied in the Fair Housing Act—included the decidedly defendant-friendly observation 
that “private policies are not contrary to the disparate-impact requirement unless they are ‘artificial, 
arbitrary, and unnecessary barriers.” Tex. Dep’t of Hous. & Cmty. Affairs v. Inclusive Cmtys. 
Project, Inc.,135 S. Ct. 2507, 2512 (2015) (quoting Griggs v. Duke Power Co., 401 U.S. 424, 431 
(1971)). 

162. See, eg., Gallagher v. Magner, 619 F.3d 823, 834 (8th Cir. 2010); Anderson v. 
Westinghouse Savannah River Co., 406 F.3d 248, 265 (4th Cir. 2005). 

163. Gulino v. N.Y. State Educ. Dep’t, 460 F.3d 361, 383 (2d Cir. 2006) (noting that hiring 
criteria are “significantly correlated with important elements of work behavior which comprise or are 
relevant to the job or jobs for which candidates are being evaluated” (quoting Albemarle Paper Co. v. 
Moody, 422 U.S. 405, 431 (1975))). 

164. Elv. Se. Pa. Transp. Auth., 479 F.3d 232, 242 (3d Cir. 2007) (quoting Lanning v. Se. Pa. 
Transp. Auth., 181 F.3d 478, 481 (3d Cir. 1999)). 

165. Michael T. Kirkpatrick, Employment Testing: Trends and Tactics, 10 EMP. RTS. & EMP. 
POL’y J. 623, 633 (2006). 

166. Id. Note, though, that this is similar to arguing that there is a less discriminatory alternative 
employment practice. This argument, then, would place the burden of the alternative employment 
practice prong on the defendant, contravening the burden-shifting scheme in the statute. See infra 
notes 170-74 and accompanying text. 

167. El, 479 F.3d at 242. 
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Putting these standards together, then, we require that employers show 
that a discriminatory hiring policy accurately—but not perfectly— 
ascertains an applicant’s ability to perform successfully the job in 
question. In addition, Title VII allows the employer to hire the 
applicant most likely to perform the job successfully over others less 
likely to do so.'® 


Thus, all circuits seem to accept varying levels of job-relatedness rather than 
strict business necessity.” 

The last piece of the disparate impact test is the “alternative employment 
practice” prong. Shortly after Griggs, the Supreme Court decided Albemarle 
Paper Co. v. Moody, holding in part that “[i]f an employer does then meet the 
burden of proving that its tests are ‘job related,’ it remains open to the 
complaining party to show that other tests or selection devices, without a 
similarly undesirable racial effect, would also serve the employer’s legitimate 
interest in ‘efficient and trustworthy workmanship.’”'” This burden-shifting 
scheme was codified in the 1991 Act as the “alternative employment practice” 
requirement.'’' Congress did not define the phrase, and its substantive meaning 


168. Id. 

169. Interestingly, it seems that many courts read identical business necessity language in the 
Americans with Disabilities Act to refer to a minimum qualification standard. See, e.g., Sullivan v. 
River Valley Sch. Dist., 197 F.3d 804, 811 (6th Cir. 1999) (“[T]here must be significant evidence that 
could cause a reasonable person to inquire as to whether an employee is still capable of performing his 
job. An employee’s behavior cannot be merely annoying or inefficient to justify an examination; 
rather, there must be genuine reason to doubt whether that employee can ‘perform job-related 
functions.” (quoting 42 U.S.C. § 12112(d)(4)(B))). Presumably, this is because disability, when 
compared to race or sex, more immediately raises questions regarding a person’s ability to perform a 
job. Ironically, however, this means that disparate impact will be more tolerated where it is less likely 
to be obviously justified. Christine Jolls has in fact argued that disparate impact is, to a degree, 
functionally equivalent to accommodations law. Jolls, supra note 90, at 652. 

170. 422 U.S. 405, 425 (1975) (quoting McDonnell Douglas Corp. v. Green, 411 U.S. 792, 801 
(1973)). 

171. 42 U.S.C. § 2000e-2(k)(1)(A) (2012). The “alternative employment practice” test has not 
always been treated as a separate step. See, e.g., Wards Cove Packing Co. v. Atonio, 490 U.S. 642, 
659 (1989) (treating the alternative employment practice test as part of the “business justification” 
phase); Dothard v. Rawlinson, 433 U.S. 321, 332 (1977) (treating the alternative employment practice 
test as a narrow tailoring requirement for the business necessity defense). The Albemarle Court, 
though creating a surrebuttal and thus empowering plaintiffs, seemed to regard the purpose of 
disparate impact as merely smoking out pretexts for intentional discrimination. 422 U.S. at 425; see 
also Primus, supra note 98, at 537. If the Albemarle Court’s approach is correct, treating the 
alternative employment practice requirement as a narrow tailoring requirement does make sense, much 
as the narrow tailoring requirement of strict scrutiny in equal protection serves the function of smoking 
out invidious purpose. City of Richmond v. J.A. Croson Co., 488 U.S. 469, 493 (1989); Rubenfeld, 
supra note 99, at 428. 

Every circuit to address the question, though, has held that the 1991 Act returned the 
doctrine to the Albemarle burden-shifting scheme. Jones v. City of Boston, 752 F.3d 38, 54 (1st Cir. 
2014); Howe v. City of Akron, 723 F.3d 651, 658 (6th Cir. 2013); Tabor v. Hilti, Inc., 703 F.3d 1206, 
1220 (10th Cir. 2013); Puffer v. Allstate Ins. Co., 675 F.3d 709, 717 (7th Cir. 2012); Gallagher v. 
Magner, 619 F.3d 823, 833 (8th Cir. 2010); Gulino v. N.Y. State Educ. Dep’t, 460 F.3d 361, 382 (2d 
Cir. 2006); Intl Bhd. of Elec. Workers Local Unions Nos. 605 & 985 v. Miss. Power & Light Co., 
442 F.3d 313, 318 (5th Cir. 2006); Anderson v. Westinghouse Savannah River Co., 406 F.3d 248, 277 
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remains uncertain. Wards Cove was the first case to use the specific phrase, so 
Congress’s instruction to reset the law to the pre-Wards Cove standard is 
particularly perplexing.” The best interpretation is most likely Albemarle’s 
reference to “other tests or selection devices, without a similarly undesirable 
racial effect.”'”* But this interpretation is slightly odd because in Albemarle, 
business necessity was still somewhat strict, and it is hard to imagine a business 
practice that is “necessary” while there exists a less discriminatory alternative 
that is just as effective.'” If business necessity or job-relatedness is a less 
stringent requirement, though, then the presence of the alternative employment 
practice requirement does at least give it some teeth. 

Now return to data mining. For now, assume a court does not apply the 
strict business necessity standard but has some variation of “job related” in 
mind (as all federal appellate courts do today).'” The threshold issue is clearly 
whether the sought-after trait—the target variable—is job related, regardless of 
the machinery used to predict it. If the target variable is not sufficiently job 
related, a business necessity defense would fail, regardless of the fact that the 
decision was made by algorithm. Thus, disparate impact liability can be found 
for improper care in target variable definition. For example, it would be 
difficult for an employer to justify an adverse determination based on the 
appearance of an advertisement suggesting a criminal record alongside the 
search results for a candidate’s name. Sweeney found such a search to have a 
disparate impact,'’° and the EEOC and several federal courts have interpreted 
Title VII to prohibit discrimination on the sole basis of criminal record, unless 
there is a specific reason the particular conviction is related to the job.'”’ This 


(4th Cir. 2005); Ass’n of Mexican-Am. Educators v. California, 231 F.3d 572, 584 (9th Cir. 2000); 
EEOC v. Joe’s Stone Crab, Inc., 220 F.3d 1263, 1275 (11th Cir. 2000); Lanning v. Se. Pa. Transp. 
Auth., 181 F.3d 478, 485 (3d Cir. 1999). The D.C. Circuit has not explicitly observed that a burden- 
shifting framework exists. 

172. Sullivan, supra note 114, at 964; Michael J. Zimmer, Individual Disparate Impact Law: 
On the Plain Meaning of the 1991 Civil Rights Act, 30 Loy. U. CHI. L.J. 473, 485 (1999). 

173. Albemarle, 422 U.S. at 425; accord, e.g., Jones, 752 F.3d at 53 (citing Albemarle to find 
meaning in the 1991 Act’s text); Allen v. City of Chicago, 351 F.3d 306, 312 (7th Cir. 2003) (same, 
but with a “see also” signal). 

174. William R. Corbett, Fixing Employment Discrimination Law, 62 SMU L. REV. 81, 92 
(2009). 

175. The difference would be whether mining for a single job-related trait, rather than a holistic 
ranking of “good employees,” is permissible at all. See infra text accompanying notes 197-99. 

176. Sweeney, supra note 41, at 51. 

177. See El v. Se. Pa. Transp. Auth., 479 F.3d 232, 243 (3d Cir. 2007) (finding that though the 
criminal record policy had a disparate impact, it satisfied business necessity in that case); Green v. Mo. 
Pac. R.R., 523 F.2d 1290, 1298 (8th Cir. 1975); McCain v. United States, No. 2:14-cv-92, 2015 WL 
1221257, at *17 (D. Vt. Mar. 17, 2015); EQUAL EMP’T OPPORTUNITY COMM’N, CONSIDERATION OF 
ARREST AND CONVICTION RECORDS IN EMPLOYMENT DECISIONS UNDER TITLE VII OF THE CIVIL 
RIGHTS ACT OF 1964 (2012), http://www.eeoc.gov/laws/guidance/upload/arrest_conviction.pdf 
[https://perma.cc/JY47-2HVT]; see also Univ. of Tex. Sw. Med. Ctr. v. Nassar, 133 S. Ct. 2517, 2540 
(2013) (“The position set out in the EEOC’s guidance and compliance manual merits respect.”); 
Michael Connett, Comment, Employer Discrimination Against Individuals with a Criminal Record: 
The Unfulfilled Role of State Fair Employment Agencies, 83 TEMP. L. REV. 1007, 1017 & nn.82-83 
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is true independent of the fact that the disparity is an artifact of third-party bias; 
all that matters is whether the target variable is job related. In the end, though, 
because determining that a business practice is not job related actually requires 
a normative determination that it is instead discriminatory, courts tend to accept 
most common business practices for which an employer has a plausible 
story. ”’ 

Once a target variable is established as job related, the first question is 
whether the model is predictive of that trait. The nature of data mining suggests 
that this will be the case. Data mining is designed entirely to predict future 
outcomes, and, if seeking a job-related trait, future job performance. One 
commentator lamented that “[f]ederal case law has shifted from a prospective 
view of meritocracy to a retrospective view, thereby weakening disparate 
impact law.”'” The author meant that, in Griggs, the Court recognized that 
education and other external factors were unequal and therefore discounted a 
measure of meritocracy that looked to past achievements, in favor of 
comparing the likelihood of future ones. But by the time the Court had decided 
Wards Cove, it had shifted to a model of retrospective meritocracy that 
presumed the legitimacy of past credentials, thus upholding the status quo.'®? 
While data mining must take the past—represented by the training data—as 
given, it generates predictions about workplace success that are much more 
accurate than predictions based on those past credentials that disparate impact 
doctrine has come to accept.'*! In a hypothetical perfect case of data mining, 
the available information would be rich enough that reliance on the past 
information would fully predict future performance. Thus, robust data mining 
would likely satisfy even the Griggs Court’s standard that the models are 
looking toward future job performance, not merely past credentials. 

The second question asks whether the model adequately predicts what it is 
supposed to predict. In the traditional context, this question arises in the case of 
general aptitude tests that might end up measuring unrelated elements of 
cultural awareness rather than intelligence.'*’ This is where the different data 


(2011) (citing EQUAL EMP’T OPPORTUNITY COMM’N, POLICY STATEMENT ON THE ISSUE OF 
CONVICTION RECORDS UNDER TITLE VII OF THE CIVIL RIGHTS ACT OF 1964 (1987), 
http:/Awww.eeoc.gov/policy/docs/convict1 .html [https://perma.cc/PY24-V8V7]). But see, e.g., Manley 
v. Invesco, 555 Fed. App’x 344, 348 (5th Cir. 2014) (per curiam) (“Persons with criminal records are 
not a protected class under Title VII.”). 

178. Michael Selmi, Was the Disparate Impact Theory a Mistake?, 53 UCLA L. REV. 701, 753 
(2006). 

179. DeSario, supra note 151, at 481. 

180. Id. at 493; see also infra Conclusion. 

181. See Don Peck, They're Watching You at Work, ATLANTIC (Nov. 20, 2013), 
http:/www.theatlantic.com/magazine/archive/2013/12/theyre-watching-you-at-work/35468 1 
[https://perma.cc/JFP8-CZKC] (discussing Google’s choice to abandon traditional hiring metrics 
because they are not good predictors of performance). 

182. See, e.g., Griggs v. Duke Power Co., 420 F.2d 1225, 1239 n.6 (4th Cir. 1970), rev'd, 401 
U.S. 424 (1971) (“Since for generations blacks have been afforded inadequate educational 
opportunities and have been culturally segregated from white society, it is no more surprising that their 
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mining mechanisms for discriminatory effects matter. Part I posited that proxy 
discrimination optimizes correctly. So if it evidences a disparate impact, it 
reflects unequal distribution of relevant traits in the real world. Therefore, 
proxy discrimination will be as good a job predictor as possible given the 
current shape of society. Models trained on biased samples and mislabeled 
examples, on the other hand, will result in correspondingly skewed assessments 
rather than reflect real-world disparities. The same effect may be present in 
models that rely on insufficiently rich or insufficiently granular datasets: by 
designation they do not reflect reality. These models might or might not be 
considered job related, depending on whether the errors distort the outcomes 
enough that the models are no longer good predictors of job performance. 

The Guidelines have set forth validation procedures intended to create a 
job-relatedness standard. Quantifiable tests that have a disparate impact must 
be validated according to the procedures in the Guidelines if possible; 
otherwise, a presumption arises that they are not job related.'® Under the 
Guidelines, a showing of validity takes one of three forms: criterion-related, 
content, or construct. Criterion-related validity “consist[s] of empirical data 
demonstrating that the selection procedure is predictive of or significantly 
correlated with important elements of job performance.”'®’ The “relationship 
between performance on the procedure and performance on the criterion 
measure is statistically significant at the 0.05 level of significance... .”'*° 
Content validity refers to testing skills or abilities that generally are or have 
been learned on the job, though not those that could be acquired in a “brief 
orientation.”'*’ Construct validity refers to a test designed to measure some 
innate human trait such as honesty. A user of a construct “should show by 
empirical evidence that the selection procedure is validly related to the 
construct and that the construct is validly related to the performance of critical 
or important work behavior(s).” "3S 

As a Statistical predictive measure, a data mining model could be 
validated by either criterion-related or construct validity, depending on the trait 
being sought. Either way, there must be statistical significance showing that the 
result of the model correlates to the trait (which was already determined to be 
an important element of job performance). This is an exceedingly low bar for 
data mining because data mining’s predictions necessarily rest on demonstrated 


performance on ‘intelligence’ tests is significantly different than whites’ than it is that fewer blacks 
have high school diplomas.”). 
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statistical relationships. Data mining will likely only be used if it is actually 
predictive of something, so the business necessity defense solely comes down 
to whether the trait sought is important enough to job performance to justify its 
use in any context. 

Even assuming the Guidelines’ validation requirement is a hurdle for data 
mining, some courts ignore the Guidelines’ recommendation that an 
unvalidated procedure be rejected, preferring to rely on “common sense” or 
finding a “manifest relationship” between the criteria and successful job 
performance.'*’ Moreover, it is possible that the Supreme Court inadvertently 
overruled the Guidelines in 2009. In Ricci v. Destefano, a case that will be 
discussed in greater detail in Part II.B, the Court found no genuine dispute that 
the tests at issue met the job-related and business necessity standards'” despite 
not having been validated under the Guidelines and despite the employer 
actively denying that they could be validated.'’' While the business necessity 
defense was not directly at issue in Ricci, “[o]n the spectrum between heavier 
and lighter burdens of justification, the Court came down decidedly in favor of 
a lighter burden.”'”” 

Thus, there is good reason to believe that any or all of the data mining 
models predicated on legitimately job-related traits pass muster under the 
business necessity defense. Models trained on biased samples, mislabeled 
examples, and limited features, however, might trigger liability under the 
alternative employment practice prong. If a plaintiff can show that an 
alternative, less discriminatory practice that accomplishes the same goals exists 
and that the employer “refuses” to use it, the employer can be found liable. In 
this case, a plaintiff could argue that the obvious alternative employment 
practice would be to fix the problems with the models. 

Fixing the models, however, is not a trivial task. For example, in the 
LinkedIn hypothetical, where the demonstrated interest in different kinds of 
employees reflects employers’ prejudice, LinkedIn is the party that determines 
the algorithm by which the discrimination occurs (in this case, based on 
reacting to third-party preferences). If an employer were to act on the 
recommendations suggested by the LinkedIn recommendation engine, there 
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would not be much he could do to make it less reflective of third-party 
prejudice, aside from calling LinkedIn and asking nicely. Thus, it could not 
really be said that the employer “refuses” to use an alternative employment 
practice. The employer could either use the third-party tool or not. Similarly, it 
might be possible to fix an app like Street Bump that suffers from reporting 
bias, but the employer would need access to the raw input data in order to do 
so.'”* In the case of insufficiently rich or granular features, the employer would 
need to collect more data in order to make the model more discerning. But 
collecting more data can be time consuming and costly,'”* if not impossible for 
legal or technical reasons. 

Moreover, the under- and overrepresentation of members of protected 
classes in data is not always evident, nor is the mechanism by which such 
under- or overrepresentation occurs. The idea that the representation of 
different social groups in the dataset can be brought into proportions that better 
match those in the real world presumes that analysts have some independent 
mechanism for determining these proportions. Thus, there are several hurdles 
to finding disparate impact liability for models employing data that under- or 
overrepresents members of protected classes. The plaintiff must prove that the 
employer created or has access to the model, can discover that there is 
discriminatory effect, and can discover the particular mechanism by which that 
effect operates. The same can be said for models with insufficiently rich feature 
sets. Clearly there are times when more features would improve an otherwise 
discriminatory outcome. But it is, almost by definition, hard to know which 
features are going to make the model more or less discriminatory. Indeed, it is 
often impossible to know which features are missing because data miners do 
not operate with causal relationships in mind. So while theoretically a less 
discriminatory alternative would almost always exist, proving it would be 
difficult. 

There is yet another hurdle. Neither Congress nor courts have specified 
what it means for an employer to “refuse” to adopt the less discriminatory 
procedure. Scholars have suggested that perhaps the employer cannot be held 
liable until it has considered the alternative and rejected it.” Thus, if the 
employer has run an expensive data collection and analysis operation without 
ever being made aware of its any discriminatory tendencies, and the employer 
cannot afford to re-run the entire operation, is the employer “refusing” to use a 
less discriminatory alternative, or does one simply not exist? How much would 
the error correction have to cost an employer before it is not seen as a refusal to 
use the procedure?'”° Should the statute actually be interpreted to mean that an 
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employer “unreasonably refuses” to use an alternative employment practice? 
These are all difficult questions, but suffice it to say, the prospect of winning a 
data mining discrimination case on alternative employment practice grounds 
seems slim. 

The third and final consideration regarding disparate impact liability for 
data mining is whether a court or Congress might reinvigorate strict business 
necessity.” In that case, things look a little better for plaintiffs bringing 
disparate impact claims. Where an employer models job tenure,'”* for example, 
a court may be inclined to hold that it is job related because the model is a 
“legitimate, non-discriminatory business objective.” But it is clearly not 
necessary to the job. The same reasoning applies to mining for any single trait 
that is job related—the practice of data mining is not focused on discovering 
make-or-break skills. Unless the employer can show that below the cut score, 
employees cannot do the work, then the strict business necessity defense will 
fail. Thus, disparate impact that occurs as an artifact of the problem- 
specification stage can potentially be addressed by strict business necessity. 

This reasoning is undermined, though, where employers do not mine for a 
single trait, but automate their decision process by modeling job performance 
on a holistic measure of what makes good employees. If employers determine 
traits of a good employee by simple ratings, and use data mining to 
appropriately divine good employees’ characteristics among several different 
variables, then the argument that the model does not account for certain skills 
that could compensate for the employee’s failings loses its force. Taken to an 
extreme, an 8,000-feature holistic determination of a “good employee” would 
still not be strictly “necessary.” Holding a business to such a standard, 
however, would simply be forbidding that business from ranking candidates if 
any disparate impact results. Thus, while the strict business necessity defense 
could prevent myopic employers from creating disparate impacts by their 
choice of target variable, it would still not address forms of data mining that 
model general job performance rather than predict specific traits. 

Disparate impact doctrine was created to address unintentional 
discrimination. But it strikes a delicate balance between allowing businesses 
the leeway to make legitimate business judgments and preventing “artificial, 
arbitrary, and unnecessary” discrimination.” Successful data mining 
operations will often both predict future job performance and have some 
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disparate impact. Unless the plaintiff can find an alternative employment 
practice to realistically point to, a tie goes to the employer. 


C. Masking and Problems of Proof 


Masking poses separate problems for finding Title VII liability. As 
discussed earlier, there is no theoretical problem with finding liability for 
masking.””! It is a disparate treatment violation as clear as any. But like 
traditional forms of intentional discrimination, it suffers from difficulties of 
proof. While finding intent from stray remarks or other circumstantial evidence 
is challenging in any scenario, masking presents additional complications for 
detection. 

Data mining allows employers who wish to discriminate on the basis of a 
protected class to disclaim any knowledge of the protected class in the first 
instance while simultaneously inferring such details from the data. An 
employer may want to discriminate by using proxies for protected classes, such 
as in the case of redlining. ” Due to housing segregation, neighborhood is a 
good proxy for race and can be used to redline candidates without reference to 
race.” This is a relatively unsophisticated example, however. It is possible that 
some combination of musical tastes,” stored “likes” on Facebook,” and 
network of friends”” will reliably predict membership in protected classes. An 
employer can use these traits to discriminate by setting up future models to sort 
by these items and then disclaim any knowledge of such proxy manipulation. 

More generally, as discussed in Part I, any of the mechanisms by which 
unintentional discrimination can occur can also be employed intentionally. The 
example described above is intentional discrimination by proxy, but it is also 
possible to intentionally bias the data collection process, purposefully mislabel 
examples, or deliberately use an insufficiently rich set of features,” though 
some of these would probably require a great deal of sophistication. These 
methods of intentional discrimination will look, for all intents and purposes, 
identical to the unintentional discrimination that can result from data mining. 
Therefore, detecting discrimination in the first instance will require the same 
techniques as detecting unintentional discrimination, namely a disparate impact 
analysis. Further, assuming there is no circumstantial evidence like an 
employer’s stray remarks with which to prove intent, a plaintiff might attempt 
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to prove intent by demonstrating that the employer is using less representative 
data, poorer examples, or fewer and less granular features than he might 
otherwise use were he interested in the best possible candidate. That is, one 
could show that the neutral employment practice is a pretext by demonstrating 
that there is a more predictive alternative. 

This looks like disparate impact analysis. A plaintiff proving masked 
intentional discrimination asks the same question as in the “alternative 
employment practice” prong: whether there were more relevant measures the 
employer could have used.” But the business necessity defense is not 
available in a disparate treatment case,” so alternative employment practice is 
not the appropriate analysis. Scholars have noted, though, that the line between 
disparate treatment and disparate impact in traditional Title VII cases is not 
always clear,” and sometimes employer actions can be legitimately 
categorized as either or both.”'' As Professor George Rutherglen has pointed 
out, “Concrete issues of proof, more than any abstract theory, reveal the 
fundamental similarity between claims of intentional discrimination and those 
of disparate impact. The evidence submitted to prove one kind of claim 
invariably can be used to support the other.””!” Rutherglen’s point is exactly 
what must happen in the data mining context: disparate treatment and disparate 
impact become essentially the same thing from an evidentiary perspective. 

To the extent that disparate impact and treatment are, in reality, different 
theories, they are often confused for each other. Plaintiffs will raise both types 
of claims as a catch-all because they cannot be sure on which theory they might 
win, so both theories will be in play in a given case.” As a result, courts often 
seek evidence of state of mind in disparate impact cases?" and objective, 
statistical evidence in disparate treatment cases.”'° Assuming the two theories 
are not functionally the same, using the same evidence for disparate treatment 
and disparate impact will only lead to more confusion and, as a result, more 
uncertainty within the courts. Thus, despite its clear nature as a theoretical 
violation, it is less clear that a plaintiff will be able to win a masking disparate 
treatment case. 

A final point is that traditionally, employers who do not want to 
discriminate go to great lengths to avoid raising the prospect that they have 
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violated the law. Thus they tend to avoid collecting information about attributes 
that reveal an individual’s membership in a protected class. Employers even 
pay third parties to collect relatively easy-to-find information on job applicants, 
such as professional honors and awards, as well as compromising photos, 
videos, or membership in online groups, so that the third party can send back a 
version of the report that “remove[s] references to a person’s religion, race, 
marital status, disability and other information protected under federal 
employment laws.””!° This allows employers to honestly disclaim any 
knowledge of the protected information. Nonetheless, if an employer seeks to 
discriminate according to protected classes, she would be able to infer class 
membership from the data. Thus, employers’ old defense to suspicion of 
discrimination—that they did not even see the information—is no longer 
adequate to separate would-be intentional discriminators from employers that 
do not intend to discriminate. 


IIL. 
THE DIFFICULTY FOR REFORMS 


While each of the mechanisms for discrimination in data mining presents 
difficulties for Title VII as currently written, there are also certain obstacles to 
reforming Title VII to address the resulting problems. Computer scientists and 
others are working on technical remedies," so to say that there are problems 
with legal remedies does not suggest that the problems with discrimination in 
data mining cannot be solved at all. Nonetheless, this Part focuses on the legal 
aspects. As it illustrates, even assuming that the political will to reform Title 
VII exists, potential legal solutions are not straightforward. 

This Part discusses two types of difficulties with reforming Title VII. 
First, there are issues internal to the data mining process that make legal reform 
difficult. For example, the subjectivity in defining a “good employee” is 
unavoidable, but, at the same time, some answers are clearly less 
discriminatory than others.”! How does one draw that line? Can employers 
gain access to the additional data necessary to correct for collection bias? How 
much will it cost them to find it? How do we identify the “correct” baseline 
historical data to avoid reproducing past prejudice or the “correct” level of 
detail and granularity in a dataset? Before laws can be reformed, policy-level 
answers to these basic technical, philosophical, and economic questions need to 
be addressed at least to some degree. 
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Second, reform will face political and constitutional constraints external 
to the logic of data mining that will affect how Title VII can be permissibly 
reformed to address it. Not all of the mechanisms for discrimination seem to be 
amenable to procedural remedies. If that holds true, only after-the-fact 
reweighting of results may be able to compensate for the discriminatory 
outcomes. This is not a matter of missing legislation; it is a matter of practical 
reality. Unfortunately, while in many cases no procedural remedy will be 
sufficient, any attempt to design a legislative or judicial remedy premised on 
reallocation of employment outcomes will not survive long in the current 
political or constitutional climate, as it raises the specter of affirmative action. 
Politically, anything that even hints at affirmative action is a nonstarter today, 
and to the extent that it is permissible to enact such policies, their future 
constitutionality is in doubt.””” 


A. Internal Difficulties 


1. Defining the Target Variable 


Settling on a target variable is a necessarily subjective exercise.” 
Disputes over the superiority of competing definitions are often insoluble 
because the target variables are themselves incommensurable. There are, of 
course, easier cases, where prejudice or carelessness leads to definitions that 
subject members of protected classes to avoidably high rates of adverse 
determinations. But most cases are likely to involve genuine business 
disagreements over ideal definitions, with each having a potentially greater or 
lesser impact on protected classes. There is no stable ground upon which to 
judge the relative merits of definitions because they often reflect competing 
ideas about the very nature of the problem at issue.””' As Professor Oscar 
Gandy has argued, “[C]ertain kind[s] of biases are inherent in the selection of 
the goals or objective functions that automated systems will [be] designed to 
support.” There is no escape from this situation; a target variable must reflect 
judgments about what really is the problem at issue in making hiring decisions. 
For certain employers, it might be rather obvious that the problem is one of 
reducing the administrative costs associated with turnover and training; for 
others, it might be improving sales; for still others, it might be increasing 
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innovation. Any argument for the superiority of one target variable over the 
other will simply make appeals to competing and incommensurate values. 

For these same reasons, however, defining the target variable also offers 
an opportunity for creative thinking about the potentially infinite number of 
ways of making sound hiring decisions. Data miners can experiment with 
multiple definitions that each seem to serve the same goal, even if these fall 
short of what they themselves consider ideal. In principle, employers should 
rely on proxies that are maximally proximate to the actual skills demanded of 
the job. While there should be a tight nexus between the sought-after features 
and these skills, this may not be possible for practical and economic reasons. 
This leaves data miners in a position to dream up many different nonideal ways 
to make hiring decisions that may have a greater or less adverse impact on 
protected classes. 

The Second Circuit considered such an approach in Hayden v. County of 
Nassau.” In Hayden, the county’s goal was to find a police entrance exam that 
was “valid, yet minimized the adverse impact on minority applicants.” The 
county thus administered an exam with twenty-five parts that could be scored 
independently. By design, a statistically valid result could be achieved by one 
of several configurations that counted only a portion of the test sections, 
without requiring all of them.”” The county ended up using nine of the sections 
as a compromise, after rejecting one configuration that was more advantageous 
to minority applicants but less statistically sound.” This is a clear example of 
defining a problem in such a way that it becomes possible to reduce the 
disparate impact without compromising the accuracy of the assessment 
mechanism. 


2. Training Data 


a. Labeling Examples 


Any solution to the problems presented by labeling must be a compromise 
between a rule that forbids employers from relying on past discrimination and 
one that allows them to base hiring decisions on historical examples of good 
employees. In theory, a rule that forbids employers from modeling decisions 
based on historical examples tainted by prejudice would address the problem of 
improper labeling. But if the only examples an employer has to draw on are 
those of past employees who had been subject to discrimination, all learned 
rules will recapitulate this discrimination. 

Title VII has always had to balance its mandate to eliminate 
discrimination in the workplace with employers’ legitimate discretion. For 
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example, one of the most common selection procedures that explicitly 
reproduced past discrimination was seniority.” Seniority was, and is still 
often, a legitimate metric for promotion and is especially important in 
collective bargaining. After the passage of Title VII, however, seniority was 
also often used to keep black people from advancing to better jobs because they 
had not been hired until Title VII forced employers to hire them.””* Despite this 
obvious problem with seniority, Title VII contains an explicit carve-out for 
“bona fide seniority or merit system[s].”””’ As a result, the Supreme Court has 
held that “absent a discriminatory purpose, the operation of a seniority system 
cannot be an unlawful employment practice even if the system has some 
discriminatory consequences.””*° Given the inherent tension between ensuring 
that past discrimination is not reproduced in future decisions and permitting 
employers legitimate discretion, it should be unsurprising that, when translated 
to data mining, the problem is not amenable to a clear solution. 

In fact, this difficulty is even more central to data mining. Data miners 
who attempt to remove the influence of prejudice on prior decisions by 
recoding or relabeling examples may find that they cannot easily resolve what 
the nonprejudicial determination would have been. As Calders and Zliobaité 
point out, “[T]he notion of what is the correct label is fuzzy.” Employers are 
unlikely to have perfectly objective and exhaustive standards for hiring; indeed, 
part of the hiring process is purposefully subjective. At the same time, 
employers are unlikely to have discriminated so completely in the past that the 
only explanation for rejecting an applicant was membership in protected 
classes. This leaves data miners tasked with correcting for prior prejudice with 
the impossible challenge of determining what the correct subjective 
employment decision would have been absent prejudice. Undoing the imprint 
of prejudice on the data may demand a complete rerendering of the biased 
decisions rather than simply adjusting those decisions according to some fixed 
statistical measure. 


b. Data Collection 


Although there are some cases with obviously skewed datasets that are 
relatively easy to identify and correct, often the source and degree of the bias 
will not be immediately apparent.” Street Bump suffered from a visually 
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evident bias when the data was plotted on a map. Boston’s Office of New 
Urban Mechanics was therefore able to partner with “a range of academics to 
take into account issues of equitable access and digital divides.””** In many 
cases, however, an analyst can only determine the extent of—and correct for— 
unintentional discrimination that results from reporting, sampling, and selection 
biases if the analyst has access to information that somehow reveals 
misrepresentations of protected classes in the dataset. Often, there may be no 
practical alternative method for collecting information that would even reveal 
the existence of a bias. 

Any attempt to correct for collection bias immediately confronts the 
problem of whether or not the employer recognizes the specific type of bias 
that is producing disparate results. Then, in order to correct for it, an employer 
must have access to the underlying data and often an ability to collect more. 
Where more data is clearly not accessible, data miners can proactively 
compensate for some of the bias by oversampling underrepresented 
communities.”** 

If the employer fails to be proactive or tries and fails to detect the bias that 
causes the disparate impact, liability is an open question. As discussed in 
Part II.B, liability partly depends on how liberally a court interprets the 
requirement that an employer “refuses” to use an alternative scheme.” Even a 
liberal interpretation, though, would require evidence of the particular type of 
discrimination at issue, coupled with evidence that such an alternative scheme 
exists. Thus, finding liability seems unlikely. Worse, where such showing is 
possible, there may be no easy or obvious way to remedy the situation. 

To address collection bias directly, an employer or an auditor must have 
access to the underlying data and the ability to adjust the model. Congress 
could require this directly of any employer using data mining techniques. Some 
employers are investing in their own data now and could potentially meet such 
requirements.”*° But employers also seem happy to rely on models developed 
and administered by third parties, who may have a far greater set of examples 
and far richer data than any individual company.” Furthermore, due to 
economies of scale that are especially important in data analysis, one can 
imagine that third parties specializing in work-force science will be able to 
offer employers this service much less expensively than they could manage it 
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themselves. If Congress attempted to demand that employers have access to the 
data, it would face strong resistance from the ever-growing data analysis 
industry, whose business depends on the proprietary nature of the amassed 
information. More likely, Congress could require audits by a third party like the 
EEOC or a private auditor, in order to protect trade secrets, but this still seems 
a tall task. Ultimately, because proactive oversampling and retroactive data 
correction are at least possible, collection bias has the most promising 
prospects for a workable remedy of any of the identified data mining 
mechanisms. 


3. Feature Selection 


Even in the absence of prejudice or bias, determining the proper degree of 
precision in the distinctions drawn through data mining can be extremely 
difficult. Under formal disparate treatment, this is straightforward: any decision 
that expressly classifies by membership in a protected class is one that draws 
distinctions on illegitimate grounds. It is far less clear, however, what 
constitutes legitimate statistical discrimination when individuation does not 
rely on proscribed criteria. In these cases, the perceived legitimacy seems to 
depend on a number of factors: (1) whether the errors seem avoidable because 
(2) gaining access to additional or more granular data would be trivial or 
(3) would not involve costs that (4) outweigh the benefits. This seems to 
suggest that the task of evaluating the legitimacy of feature selection can be 
reduced to a rather straightforward cost-benefit analysis. Companies would 
have an obligation to pursue ever more—and more granular—data until the 
costs of gathering that data exceed the benefits conferred by the marginal 
improvements in accuracy. 

Unfortunately, as is often the case with cost-benefit analyses, this 
approach fails to consider how different actors will perceive the value of the 
supposed benefits as well as the costs associated with errors. The obvious 
version of this criticism is that “actuarially saddled” victims of inaccurate 
determinations may find cold comfort in the fact that certain decisions are 
rendered more reliably overall when decision makers employ data mining.”** A 
more sophisticated version of this criticism focuses on the way such errors 
assign costs and benefits to different actors at systematically different rates. A 
model with any error rate that continues to turn a profit may be acceptable to 
decision makers at a company, no matter the costs or inconvenience to specific 
customers.”” Even when companies are subject to market pressures that would 
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force them to compete by lowering these error rates, the companies may find 
that there is simply no reason to invest in efforts that do so if the errors happen 
to fall disproportionately on especially unprofitable groups of consumers. 
Furthermore, assessing data mining as a matter of balancing costs and benefits 
leaves no room to consider morally salient disparities in the degree to which 
the costs are borne by different social groups. This raises the prospect that there 
might be systematic differences in the rates at which members of protected 
classes are subject to erroneous determinations.” Condemning these groups to 
bear the disproportionate burden of erroneous determinations would strike 
many as highly objectionable, despite greater accuracy in decision making for 
the majority group.”*' Indeed, simply accepting these cost differences as a 
given would subject those already in less favorable circumstances to less 
accurate determinations. 

Even if companies assume the responsibility for ensuring that members of 
protected classes do not fall victim to erroneous determinations at 
systematically higher rates, they could find that increasing the resolution and 
range of their analyses still fails to capture the causal relationships that account 
for different outcomes because those relationships are not easily represented in 
data.” In such cases, rather than reducing the error rate for those in protected 
classes, data miners could structure their analyses to minimize the difference in 
error rates between groups. This solution may involve some unattractive 
tradeoffs, however. In reducing the disparate impact of errors, it may increase 
the overall amount of errors. In other words, generating a model that is equally 
unfair to protected and unprotected classes might increase the overall amount 
of unfairness. 


4. Proxies 


Computer scientists have been unsure how to deal with redundant 
encodings in datasets. Simply withholding these variables from the data mining 
exercise often removes criteria that hold demonstrable and justifiable relevance 
to the decision at hand. As Calders and Zliobaité note, “[I]t is problematic [to 
remove a correlated attribute] if the attribute to be removed also carries some 
objective information about the label [quality of interest].”’* Part of the 
problem seems to be that there is no obvious way to determine how correlated a 
relevant attribute must be with class membership to be worrisome. Nor is there 
a self-evident way to determine when an attribute is sufficiently relevant to 
justify its consideration, despite its high correlation with class membership. As 
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Professors Devin Pope and Justin Sydnor explain, “[VJariables are likely 
neither solely predictive nor purely proxies for omitted characteristics.” 

But there is a bigger problem here: attempting to ensure fairly rendered 
decisions by excising highly correlated criteria only makes sense if the 
disparate impact happens to be an avoidable artifact of a particular way of 
rendering decisions. And yet, even when denied access to these highly 
correlated criteria, data mining may suggest alternative methods for rendering 
decisions that still result in the same disparate impact. Focusing on isolated 
data points may be a mistake because class membership can be encoded in 
more than one specific and highly correlated criterion. Indeed, it is very likely 
that class membership is reflected across a number of interrelated data 
points.” But such outcomes might instead demonstrate something more 
unsettling: that other relevant criteria, whatever they are, happen to be 
possessed at different rates by members of protected classes. This explains 
why, for instance, champions of predictive policing have responded to critics 
by arguing that “[i]f you wanted to remove everything correlated with race, you 
couldn’t use anything. That’s the reality of life in America.”” Making 
accurate determinations means considering factors that are somehow correlated 
with proscribed features. 

Computer scientists have even shown that “[r]emoving all such correlated 
attributes before training does remove discrimination, but with a high cost in 
classifier accuracy.” This reveals a rather uncomfortable truth: the current 
distribution of relevant attributes—attributes that can and should be taken into 
consideration in apportioning opportunities fairly—is demonstrably correlated 
with sensitive attributes because the sensitive attributes have meaningfully 
conditioned what relevant attributes individuals happen to possess.” As such, 
attempts to ensure procedural fairness by excluding certain criteria from 
consideration may conflict with the imperative to ensure accurate 
determinations. The only way to ensure that decisions do not systematically 
disadvantage members of protected classes is to reduce the overall accuracy of 


244. Devin G. Pope & Justin R. Sydnor, Implementing Anti-Discrimination Policies in 
Statistical Profiling Models, 3 AM. ECON. J. 206, 206 (2011). 

245. Supra discussion accompanying note 101. 

246. Labi, supra note 5 (quoting Ellen Kurtz, Director of Research for Philadelphia’s Adult 
Probation and Parole Department). 

247. Toon Calders & Sicco Verwer, Presentation at the European Conference on Machine 
Learning and Principles and Practice of Knowledge Discovery in Databases: Three Naïve Bayes 
Approaches for Discrimination-Free Classification 9 (2010), 
http://wwwis.win.tue.nl/~tcalders/dadm/lib/exe/fetch.php?media=ecmlpkdd_2010_discrimination.pdf 
[http://perma.cc/9V72-2NVM]. 

248. Ina sense, computer scientists have unwittingly furnished the kind of evidence that social 
scientists routinely seek: the particular contours of inequality. See, e.g., SOCIAL INEQUALITY (Kathryn 
M. Neckerman ed., 2004). 


722 CALIFORNIA LAW REVIEW [Vol. 104:671 


all determinations. As Dwork et al. remark, these results “demonstrate a 
quanti[t]ative trade-off between fairness and utility.” ® 

In certain contexts, data miners will never be able to fully disentangle 
legitimate and proscribed criteria. For example, the workforce optimization 
consultancy, Evolv, discovered that “[d]istance between home and work ... is 
strongly associated with employee engagement and retention.””® Despite the 
strength of this finding, Evolv “never factor[s] [it] into the score given each 
applicant... because different neighborhoods and towns can have different 
racial profiles, which means that scoring distance from work could violate 
equal-employment-opportunity standards.””°! Scholars have taken these cases 
as a sign that the “major challenge is how to find out which part of information 
carried by a sensitive (or correlated) attribute is sensitive and which is 
objective.” While researchers are well aware that this may not be easy to 
resolve, let alone formalize into a computable problem, there is a bigger 
challenge from a legal perspective: any such undertaking would necessarily 
wade into the highly charged debate over the degree to which the relatively less 
favorable position of protected classes warrants the protection of 
antidiscrimination law in the first instance. 

The problems that render data mining discriminatory are very rarely 
amenable to obvious, complete, or welcome resolution. When it comes to 
setting a target variable and feature selection, policy cannot lay out a clear path 
to improvement; reducing the disparate impact will necessitate open-ended 
exploration without any way of knowing when analysts have exhausted the 
possibility for improvement. Likewise, policies that compel institutions to 
correct tainted datasets or biased samples will make impossible demands of 
analysts. In most cases, they will not be able to determine what the objective 
determination should have been or independently observe the makeup of the 
entire population. Dealing with both of these problems will ultimately fall to 
analysts’ considered judgment. Solutions that reduce the accuracy of decisions 
to minimize the disparate impact caused by coarse features and unintentional 
proxies will force analysts to make difficult and legally contestable trade-offs. 
General policies will struggle to offer the specific guidance necessary to 
determine the appropriate application of these imperfect solutions. And even 
when companies voluntarily adopt such strategies, these internal difficulties 
will likely allow a disparate impact to persist. 
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B. External Difficulties 


Assuming the internal difficulties can be resolved, there are further 
political and constitutional restraints on addressing Title VII’s inadequacies 
with respect to data mining. Data mining discrimination will force a 
confrontation between the two divergent principles underlying 
antidiscrimination law: anticlassification and antisubordination.”* Which of 
these two principles motivates discrimination law is a contentious debate, and 
making remedies available under antidiscrimination law will require a 
commitment to antisubordination principles that have thus far not been 
forthcoming from legislatures. This is not merely a political concern, as 
substantive remediation is becoming ever more suspect constitutionally as 
well.” While such remedies may be politically and legally impossible, the 
nature of data mining itself makes them practically necessary. Accordingly, 
these external difficulties may prevent antidiscrimination law from fully 
addressing data mining discrimination. 

Two competing principles have always undergirded antidiscrimination 
law: anticlassification and antisubordination. Anticlassification is the narrower 
of the two, holding that the responsibility of the law is to eliminate the 
unfairness individuals in certain protected classes experience due to decision 
makers’ choices.” Antisubordination theory, in contrast, holds that the goal of 
antidiscrimination law is, or at least should be, to eliminate status-based 
inequality due to membership in those classes, not as a matter of procedure, but 
of substance.”*° 

Different mitigation policies effectuate different rationales. Disparate 
treatment doctrine arose first, clearly aligning with the anticlassification 
principle by proscribing intentional discrimination, in the form of either 
explicit singling out of protected classes for harm or masked intentional 
discrimination. Since disparate impact developed, however, there has never 
been clarity as to which of the principles it is designed to effectuate.” On the 
one hand, disparate impact doctrine serves anticlassification by being an 
“evidentiary dragnet” used to “smoke out” well-hidden disparate treatment.” 
On the other hand, as an effects-based doctrine, there is good reason to believe 
it was intended to address substantive inequality.” In this sense, the “business 
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necessity” defense is a necessary backstop that prevents members of 
traditionally disadvantaged groups from simply forcing their way in without the 
necessary skills or abilities.” 

Thus, the mapping from anticlassification and antisubordination to 
disparate treatment and disparate impact was never clean. Early critics of civil 
rights laws actually complained that proscribing consideration of protected 
class was a subsidy to black people.”*' This argument quickly gave way in the 
face of the rising importance of the anticlassification norm.” Over the years, 
the anticlassification principle has come to dominate the landscape so 
thoroughly that a portion of the populace thinks (as do a few Justices on the 
Supreme Court) that it is the only valid rationale for antidiscrimination law. 

The move away from antisubordination began only five years after 
disparate impact was established in Griggs. In Washington v. Davis, the Court 
held that disparate impact could not apply to constitutional claims because 
equal protection only prohibited intentional discrimination.” Since then, the 
various affirmative action cases have overwritten the distinction between 
benign and harmful categorizations of race in favor of a formalistic 
anticlassification principle, removed from its origins as a tool to help members 
of historically disadvantaged groups.” White men can now bring disparate 
treatment claims.*” If antidiscrimination law is no longer thought to serve the 
purpose of improving the relative conditions of traditionally disadvantaged 
groups, antisubordination is not part of the equation. 

While the Court has clearly established that antisubordination is not part 
of constitutional equal protection doctrine, that it does not mean that 
antisubordination cannot animate statutory antidiscrimination law. 
Antisubordination and anticlassification came into sharp conflict in Ricci v. 
Destefano, a 2009 case in which the City of New Haven refused to certify a 
promotion exam given to its firefighters on the grounds that it would have 
produced a disparate impact based on its results.°°’ The Supreme Court held 
that the refusal to certify the test, a facially race-neutral attempt to correct for 
perceived disparate impact, was in fact a race-conscious remedy that 
constituted disparate treatment of the majority-white firefighters who would 
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have been promoted based on the exam’s results.” The Court held that 
disparate treatment cannot be a remedy for disparate impact without a “strong 
basis in evidence” that the results would lead to actual disparate treatment 
liability.” 

Ricci was the first indication at the Supreme Court that disparate impact 
doctrine could be in conflict with disparate treatment.” The Court had 
previously ruled in essence that the antisubordination principle could not 
motivate a constitutional decision,’ but it had not suggested that law 
effectuating that principle could itself be discriminatory against the dominant 
groups. That has now changed.”” 

The decision has two main consequences for data mining. First, where the 
internal difficulties in resolving discrimination in data mining described above 
can be overcome, legislation that requires or enables such resolution may run 
afoul of Ricci. Suppose, for example, Congress amended Title VII to require 
that employers make their training data and models auditable. In order to 
correct for detected biases in the training data that result in a model with a 
disparate impact, the employer would first have to consider membership in the 
protected class. The remedy is inherently race-conscious. The Ricci Court did 
hold that an employer may tweak a test during the “test-design stage,” 
however.””* So, as a matter of timing, data mining might not formally run into 
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Ricci if the bias resulting in a disparate impact is corrected before applied to 
individual candidates. After an employer begins to use the model to make 
hiring decisions, only a “strong basis in evidence” that the employer will be 
successfully sued for disparate impact will permit corrective action.’” Of 
course, unless every single model used by an employer is subject to a 
prescreening audit (an idea that seems so resource intensive that it is effectively 
impossible), the disparate impact will be discovered only when the employer 
faces complaints. Additionally, while Ricci’s holding was limited in scope, the 
“strong basis in evidence” standard did not seem to be dictated by the logic of 
the opinion, which illustrated a more general conflict between disparate 
treatment and disparate impact.” 

Second, where the internal difficulties cannot be overcome, there is likely 
no way to correct for the discriminatory outcomes aside from results-focused 
balancing, and requiring this will pose constitutional problems. For those who 
adhere to the anticlassification principle alone, such an impasse may be 
perfectly acceptable. They might say that as long as employers are not 
intentionally discriminating based on explicitly proscribed criteria, the chips 
should fall where they may. To those who believe some measure of substantive 
equality is important over and above procedural equality, this result will be 
deeply unsatisfying. 

An answer to the impasse created by situations that would require results- 
focused rebalancing is to reexamine the purpose of antidiscrimination law. The 
major justification for reliance on formal disparate treatment is that prejudice is 
simply irrational and thus unfair. But if an employer knows that his model has 
a disparate impact, but it is also his most predictive, the argument that the 
discrimination is irrational loses any force. Thus, data mining may require us to 
reevaluate why and whether we care about not discriminating. 

Consider another example involving tenure predictions, one in which an 
employer ranks potential employees with the goal of hiring only those 
applicants that the company expects to retain for longer periods of time. In 
optimizing its selection of applicants in this manner, the employer may 
unknowingly discriminate against women if the historical data demonstrates 
that they leave their positions after fewer years than their male counterparts. If 
gender accounts for a sufficiently significant difference in employee tenure, 
data mining will generate a model that simply discriminates on the basis of 
gender or those criteria that happen to be proxies for gender. Although 
selecting applicants with an eye to retention might seem both rational and 
reasonable, granting significance to predicted tenure would subject women to 
systematic disadvantage if gender accounts for a good deal of the difference in 
tenure. If that is the case, any data mining exercise that attempts to predict 
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tenure will invariably rediscover this relationship. One solution could be for 
Congress to amend Title VII to reinvigorate strict business necessity.””° This 
would allow a court to accept that relying on tenure is rational but not strictly 
“necessary” and that perhaps other factors could make up for the lack of 
predicted tenure. 

But this solution and all others must rely on the antisubordination 
principle. Consider this question: should the law permit a company to hire no 
women at all—or none that it correctly predicts will depart following the birth 
of a child—because it is the most rational choice according to their model?””’ 
The answer seems obviously to be no. But why not? What forms the basis for 
law’s objection to rational decisions, based on seemingly legitimate criteria, 
that place members of protected classes at systematic disadvantage? The 
Supreme Court has observed that, “Title VII requires employers to treat their 
employees as individuals, not ‘as simply components of a racial, religious, 
sexual, or national class.’””’* On the strength of that statement, the Court held 
that employers could not force women to pay more into an annuity because 
they, as women, were likely to live longer.” But it is not clear that this 
reasoning translates directly to data mining. Here, the model takes a great deal 
of data about an individual, and while it does make a determination based on 
statistics, it will make a different one if analyzing two different women. So if 
the model said to hire no women, it would be illegal, but, according to the 
doctrine, perhaps only because every woman ends up with the same result. 

The only escape from this situation may be one in which the relevance of 
gender in the model is purposefully ignored and all factors correlated with 
gender are suppressed. The outcome would be a necessarily less accurate 
model. The justification for placing restrictions on employers, and limiting the 
effectiveness of their data mining, would have to depend on an entirely 
different set of arguments than those advanced to explain the wrongfulness of 
biased data collection, poorly labeled examples, or an impoverished set of 
features. Here, shielding members of protected classes from less favorable 
treatment is not justified by combatting prejudice or stereotyping. In other 
words, any prohibition in this case could not rest on a procedural commitment 
to ensuring ever more accurate determinations. Instead, the prohibition would 
have to rest on a substantive commitment to equal representation of women in 
the workplace. That is, it would have to rest on a principle of antisubordination. 
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The dilemma is clear: the farther the doctrine gets from substantive 
remediation, the less utility it has in remedying these kinds of discriminatory 
effects.°° But the more disparate impact is thought to embody the 
antisubordination principle—as opposed to the “evidentiary dragnet” in service 
of the anticlassification norm—the more it will invite future constitutional 
challenges.”*! 

This also raises a point about disparate treatment and data mining. Within 
data mining, the effectiveness of prohibiting the use of certain information 
exists on a spectrum. On one end, the prohibition has little to no effect because 
either the information is redundantly encoded or the results do not vary along 
lines of protected class. On the other end, the prohibition reduces the accuracy 
of the models. That is, if protected class data were not prohibited, that 
information would alter the results, presumably by making members of 
protected classes worse (or, in some cases, better) off. Thus, as a natural 
consequence of data mining, a command to ignore certain data has either no 
effect’ or the effect of altering the fortunes of those protected classes in 
substantive ways. Therefore, with respect to data mining, due to the zero-sum 
nature of a ranking system, even disparate treatment doctrine is a reallocative 
remedy similar to affirmative action.” Once again, this erodes the legitimate 
rationale for on the one hand supporting an anticlassification principle but on 
the other, holding fast against antisubordination in this context. The two 
principles tend to accomplish the same thing, but one is less effective at 
achieving substantive equality. 

This reveals that the pressing challenge does not lie with ensuring 
procedural fairness through a more thorough stamping out of prejudice and bias 
but rather with developing ways of reasoning to adjudicate when and what 
amount of disparate impact is tolerable. Abandoning a belief in the efficacy of 
procedural solutions leaves policy makers in an awkward position because 
there is no definite or consensus answer to questions about the fairness of 
specific outcomes. These need to be worked out on the basis of different 
normative principles. At some point, society will be forced to acknowledge that 
this is really a discussion about what constitutes a tolerable level of disparate 
impact in employment. Under the current constitutional order and in the 
political climate, it is tough to even imagine having such a conversation. But, 
until that happens, data mining will be permitted to exacerbate existing 
inequalities in difficult-to-counter ways. 
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CONCLUSION 


This Essay has identified two types of discriminatory outcomes from data 
mining: a family of outcomes where data mining goes “wrong” and outcomes 
where it goes too “right.” Data mining can go wrong in any number of ways. It 
can choose a target variable that correlates to protected class more than others 
would, reproduce the prejudice exhibited in the training examples, draw 
adverse lessons about protected classes from an unrepresentative sample, 
choose too small a feature set, or not dive deep enough into each feature. Each 
of these potential errors is marked by two facts: the errors may generate a 
manifest disparate impact, and they may be the result of entirely innocent 
choices made by data miners. 

Where data mining goes “right,” data miners could not have been any 
more accurate given the starting point of the process. This very accuracy, 
exposing an uneven distribution of attributes that predict the target variable, 
gives such a result its disparate impact. If the data accurately models inequality, 
attempts to devise an alternative way of making the same prediction will only 
narrow the disparate impact if these efforts reduce the accuracy of the decision 
procedure. By now, it should be clear that Title VII, and very likely other 
similarly process-oriented civil rights laws, cannot effectively address this 
situation. 

This means something different for the two families, and it should be 
slightly more surprising for the former. At a high level of abstraction, where a 
decision process goes “wrong” and this wrongness creates a disparate impact, 
Title VII and similar civil rights laws should be up to the task of solving the 
problem; that is ostensibly their entire purpose. But aside from a few more 
obvious cases involving manifest biases in the dataset, it is quite difficult to 
determine ahead of time what “correct” data mining looks like. A decision 
maker can rarely discover that the choice of a particular target variable is more 
discriminatory than other choices until after the fact, at which point it may be 
difficult and costly to change course. While data miners might have some 
intuitions about the influence that prejudice or bias played in the prior decisions 
that will serve as training data, data miners may not have any systematic way 
of measuring and correcting for that influence. And even though ensuring 
reliable samples before training a model is a possibility, the data may never be 
perfect. It may be impossible to determine, ex ante, how much the bias 
contributes to the disparate impact, it may not be obvious how to collect 
additional data that makes the sample more representative, and it may be 
prohibitively expensive to do so. Companies will rarely be able to resolve these 
problems completely; their models will almost always suffer from some 
deficiency that results in a disparate impact. A standard that holds companies 
liable for any amount of theoretically avoidable disparate impact is likely to 
ensnare all companies. Thus, even at this level of abstraction, it becomes clear 
that holding the decision makers responsible for these disparate impacts is at 
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least partly troubling from a due process perspective. Such concerns may 
counsel against using data mining altogether. This would be a perverse 
outcome, given how much even imperfect data mining can do to help reduce 
the very high rates of discrimination in employment decisions. 

If liability for getting things “wrong” is difficult to imagine, how does 
liability for getting things “right”? make any more sense? That proxy 
discrimination largely rediscovers preexisting inequalities suggests that perhaps 
Title VII is not the appropriate remedial vehicle. If what is at stake are the 
results of decades of historical discrimination and wealth concentration that 
have created profound inequality in society, is that not too big a problem to 
remedy through individual lawsuits, assuming affirmative action and similar 
policies are off the table? Thus, perfect data mining forces the question: if 
employers can say with certainty that, given the status quo,” candidates from 
protected classes are on average less ready for certain jobs than more privileged 
candidates, should employers specifically be penalized for hiring fewer 
candidates from protected classes? 

Doctrinally, the answer is yes, to some extent. Professor Christine Jolls 
has written that disparate impact doctrine is akin to accommodation in 
disability law—that is, both accommodations and disparate impact specifically 
require employers to depart from pure market rationality and incur costs 
associated with employing members of protected classes.” Similarly, the Title 
VII annuity cases? and Title VII’s ban on following racist third-party 
preferences” each require a departure from market rationality. Thus, Title VII 
makes that decision to a degree. But to what degree? How much cost must an 
employer bear? 

Title VII does not require an employer to use the least discriminatory 
means of running a business.” Likewise, Title VII does not aim to remedy 
historical discrimination and current inequality by imposing all the costs of 
restitution and redistribution on individual employers.” It is more 
appropriately understood as a standard of defensible disparate impact. One 
route, then, to addressing the problems is to make the inquiry more searching 
and put the burden on the employer to avoid at least the easy cases. In a system 
that is as unpredictable as data mining can be, perhaps the proper way of 


284. We cannot stress enough the import of these caveats. Certainty is a strong and unlikely 
precondition, and the status quo should not be taken as a given, as we explain below. 
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thinking about the solution is a duty of care, a theory of negligent 
discrimination.” 

But if Title VII alone cannot solve these problems, where should society 
look for answers? Well, the first answer is to question the status quo. Data 
mining takes the existing state of the world as a given and ranks candidates 
according to their predicted attributes in that world. Data mining, by its very 
nature, treats the target variable as the only item that employers are in a 
position to alter; everything else that happens to correlate with different values 
for the target variable is assumed stable. But there are many reasons to question 
these background conditions. Sorting and selecting individuals according to 
their apparent qualities hides the fact that the predicted effect of possessing 
these qualities with respect to a specific outcome is also a function of the 
conditions under which these decisions are made. Recall the tenure example 
from Part III.B. In approaching appropriate hiring practices as a matter of 
selecting the “right” candidates at the outset, an employer will fail to recognize 
potential changes that he could make to workplace conditions. A more family- 
friendly workplace, greater on-the-job training, or a workplace culture more 
welcoming to historically underrepresented groups could affect the course of 
employees’ tenure and their long-term success in ways that undermine the 
seemingly prophetic nature of data mining’s predictions. 

These are all traditional goals for reducing discrimination within the 
workplace, and they continue to matter even in the face of the eventual 
widespread adoption of data mining. But data can play a role here, too. For 
example, comparing the performance of equally qualified candidates across 
different workplaces can help isolate the formal policies and institutional 
dynamics that are more or less likely to help workers flourish. Research of this 
sort could also reveal areas for potential reform.””! 

Education is also important. Employers may take some steps to rectify the 
problem on their own if they better understand the cause of the disparity. Right 
now, many of the problems described in Part I are relatively unknown. But the 
more employers and data miners understand these pitfalls, the more they can 
strive to create better models on their own. Many employers switch to data- 
driven practices for the express purpose of eradicating bias;” if employers 
discover that they are introducing new forms of bias, they can correct course. 

Even employers seeking only to increase efficiency or profit may find that 
their incentives align with the goals of nondiscrimination. Faulty data and data 
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mining will lead employers to overlook or otherwise discount people who are 
actually “good” employees. Where the cost of addressing these problems is at 
least compensated for by a business benefit of equal or greater value, 
employers may have natural incentives to do so. 

Finally, employers could also make more effective use of the tools that 
computer scientists have begun to develop.” Advances in these areas will 
depend, crucially, on greater and more effective collaboration between 
employers, computer scientists, lawyers, advocates, regulators, and policy 
makers.” 

This Essay is a call for caution in the use of data mining, not its 
abandonment. While far from a panacea, data mining can and should be part of 
a panoply of strategies for combatting discrimination in the workplace and for 
promoting fair treatment and equality. Ideally, institutions can find ways to use 
data mining to generate new knowledge and improve decision making that 
serves the interests of both decision makers and protected classes. But where 
data mining is adopted and applied without care, it poses serious risks of 
reproducing many of the same troubling dynamics that have allowed 
discrimination to persist in society, even in the absence of conscious prejudice. 


293. See list supra note 217. 
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Silicon Valley is stumped: Even 
A.I. cannot always remove bias 
from hiring 


e Al software to eliminate the prejudice of human hiring managers has produced 
encouraging early results at corporations. 


e But tech executives with experience at Google, Microsoft and Facebook say the 
algorithmic revolution in hiring is moving too fast. 


e Algorithm auditing firms want to see the code; public policy exsarts want to press 
governments to forcethe algorithms into the open before,its,toe late,they say. CNBC TV 


Eric Rosenbaum | @erprose 
Published 9:43 AM ET Wed, 30 May 2018 | Updated 5:20 PM ET Wed, 30 May 2018 


At a recent MIT event on the future of work in New York City for its 

high-achieving alumni network, Andrew McAfee, co-director of TRENDING NOW 
MIT's Initiative on the Digital Economy and a principal research 

scientist at the university's Sloan School of Management, said 

leaders are realizing that a lot of their human practices, human 

resources and human capit 
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8/10/2018 Silicon Valley is stumped, even A.I. cannot remove bias from hiring 


McAfee's view: "If you want the bias out, get the algorithms in." 


Silicon Valley is investing in many start-ups selling the idea that they 
can solve the problem of human bias in job-hiring decisions with 
artificial intelligence. But a new class of independent algorithm 
auditing firms and public policy experts — with experience at some 
of the largest tech companies in the world and educations from elite 
institutions — say ‘algorithmic bias' has already been proved to exist 
in other areas. As a result, the rapid uptake of Als for hiring in the 
market has moved too fast, and with too little scrutiny, they say. 


Play games, get hired, 


Source: Pymetrics 


Algorithms can help HR professionals make smart hiring decisions, 
but these algorithms can often be biased against minorities, said 
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8/10/2018 Silicon Valley is stumped, even A.I. cannot remove bias from hiring 
human bias influenced the algorithm, and it's up to humans to notice 
the bias and fix it. 


Traditional résumé review leads to women and minorities being at a 
50 percent to 67 percent disadvantage, according to start-up 
pymetrics, which attempts to go well beyond the résumé in assessing 
job applicants using neuroscience games and AI. 


Companies using AI can reduce those figures dramatically, 
pymetrics said, as long as the input data is accurate and remains 
unbiased. 


That's a big "if." 


Al can work, ‘as long as' the input data is accurate 






Cathy O'Neil, whg also spoke at the MIT future-of-werk event, said ; = 
the hiring algorithmas:now coming into the humanueseunees field are CNBC TV MENU 
a perfect test case for her skepticism about the tech utopian 


movement, and she uses these job algorithms often in presentations. 


O'Neil, an academically trained mathematician who studied and 

worked at UC Berkeley, Harvard and MIT — and left a job on Wall 

Street to join the Occupy Wall Street movement and write a book on 

the dangers of algorithms — often employs a thought experiment in 

talks she gives: Imagine what a machine-learning hiring algorithm 

trained on Fox News data would result in, even if reasonable choices 

were being made by the data science team. Then she points out that 

it doesn't have to be an outrageous example like Fox News, because 

there is no perfect workplace with perfect hiring policies, perfect Q 


https://www.cnbc.com/2018/05/30/silicon-valley-is-stumped-even-a-i-cannot-remove-bias-from-hiring.html 3/14 


8/10/2018 Silicon Valley is stumped, even A.I. cannot remove bias from hiring 
raises and promotion methods, and a culture that welcomes all 
people equally. 


"When we blithely train algorithms on historical 
data, to a large extent we are setting ourselves 
up to merely repeat the past. ... We'll need to do 
more, which means examining the bias 
embedded in the data." 


-Cathy O'Neil, author of "Weapons of Math Destruction: How Big Data Increases Inequality and 


Threatens Democracy" 


"It's going to take some real thought," said Dr. Lori Kletzer, an 
economics professor at Colby College in an interview with CNBC. 
e the 
implications are societal, sowie.canit just 





"It's not just going tg happen. And it's important to rais 





questions now. .ma 
leave it to the market, because the market only cares about the 
bottom line." 


To start, the makeup of the tech industry creating the hiring 
algorithms isn't perfect. While Silicon Valley has a long history of 
encouraging immigrant entrepreneurs and bringing in foreign 
workers from around the world on skill visas, it has been criticized 
for diversity in terms of hiring from within the national population. 


A report from the federal government's Government Accountability 
Office released in November 2017 found that the technology 
industry is behind other sectors in diversity of its workforce. "The 


O 
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estimated percentage of minority technology workers increased 
from 2005 to 2015, but GAO found that no growth occurred for 
female and black workers, whereas Asian and Hispanic workers 
made statistically significant increases. Further, female, black and 
Hispanic workers remain a smaller proportion of the technology 
workforce — mathematics, computing and engineering occupations 
— compared to their representation in the general workforce." 


"When we blithely train algorithms on historical data, to a large 
extent we are setting ourselves up to merely repeat the past. If we 
want to get beyond that, beyond automating the status quo, we'll 
need to do more, which means examining the bias embedded in the 
data. The data is, after all, simply a reflection of our imperfect 
culture," O'Neil, who now runs her own algorithm auditing firm, 
said via email. 





erking CNBC TV MENU 


The tradition fob application process,isn't. 
Dr. Frida Polli, pymetrics CEO and co-founder, also has an 
extensive academic résumé, which includes an MBA from Harvard 
and a postdoctoral fellowship in neuroscience from MIT. Though 
despite the impressive accomplishments, she feels that simply listing 
them on her résumé didn't provide employers much information 
about her potential. 


Pymetrics is working with companies such as Unilever, Accenture, 
LinkedIn and Tesla. The company uses behavioral neuroscience and 
artificial intelligence to help identify candidates in a more predictive 
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and unbiased way. Pymetrics bypasses the résumé, using data 
generated from brain games to match applicants with roles. 





Ah, This is how Al is changing the way 
> you apply for jobs 


2:07 PM ET Tue, 13 March 2018 | 04:40 


HireVue is anothe!$tart-up working with the corpordtelindustrial 

. MARKETS BUSINESS NEWS 
psychologists to make sure employer assessment tools are up to 
industry standards and, by adding AI to the mix, eliminating bias. It 
has been around for more than a decade, starting with tech that 
allowed for video interviews and moving more recently to Al-based 


job assessments. 


"We can measure it, unlike the human mind, where we can't see 
what they're thinking or if they're systematically biased," Lindsey 
Zuloaga, director of data science at HireVue, recently told CNBC. 


Once the candidate reaches a human recruiter, companies using 
HireVue have reported a much more diverse candidate pool: 
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Unilever has improved the diversity of its talent pool by 16 percent 
since partnering with HireVue. "If the team does notice a skew in 
results, it can evaluate the algorithm to see what went wrong and 
remove the bad data," Zuloaga said. 


"Al is not impartial or neutral," said Meredith Whittaker, co-founder 
of the AI Now Institute at New York University, and founder of 
Google's Open Research group. AI Now — which Whittaker co- 
founded with Kate Crawford, an NYU professor and principal 
researcher at Microsoft Research — aims to move beyond what it 
describe as "minimal oversight" of AI. Algorithmic bias is one of its 
core research areas. 


"In the case of systems meant to automate candidate search and 
hiring, we need to ask ourselves: What assumptions about worth, 
oduce? Who 






ability and potential do these systems reflect and rep 


was at the table whenthese assumptions were encoders" Whittaker ei) = 
asked. 
More from @Work: 


There are two classes of workers in Silicon Valley 
Co-working spaces for women rise as response to #MeToo 
A new robot poised to make coffee better than a barista 


Whittaker said HireVue, for instance, creates models based on "top 
performers" at a firm, then uses emotion detection systems that pick 
up cues from the human face to evaluate job applicants based on 
these models. "This is alarming, because firms that are using such 
software may not have diverse workforces to begin with, and often 
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have decreasing diversity at the top. And given that systems like 
HireVue are proprietary and not open to review, how do we validate 
their claims to fairness and ensure that they aren't simply tech- 
washing and amplifying longstanding patterns of discrimination?" 


In a statement to CNBC, Loren Larsen, CTO of HireVue, said, "It is 
extremely important to audit the algorithms used in hiring to detect 
and correct for any bias. ... No company doing this kind of work 
should depend only on a third-party firm to ensure that they are 
doing this work in a responsible way. Third parties can be very 
helpful, and we have sought out third-party data-science experts to 
review our algorithms and methods to ensure they are state-of-the- 
art. However, it's the responsibility of the company itself to audit the 
algorithms as an ongoing, day-to-day process." 





drastic and harmful downs! de of Al o = 
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N : MARKETS . BUSINESS NEWS : CNBC TV MENU 
Pymetrics said the biggest hurdle with HR teams within corporations 


The potential ' 


are legal concerns about bias. That's why pymetrics developed a 
process to de-bias their algorithms and has open-sourced that 
methodology on GitHub. It "wants all companies, regardless of 
industry, to have the tools to detect and remove bias from their 
algorithms," Polli said. But it does not let third-party algorithm 
auditing firms, like O'Neil's, review its actual job-hiring code for 
undetected bias. 


"The algorithms themselves are not the solution, because they could 
actually make it worse. Audited algorithms that are shown to be free 
of gender bias ... are the answer to removing bias," Polli said. She 
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added, "If you want to have a third-party auditor, fantastic. But the 
most critical thing is that it is being done however it is getting done." 


Polli said the pymetrics process has now 50,000 pieces of data 
tested, and that does give it confirmation of no gender or ethnic bias. 


That kind of confidence in an internal review process doesn't sit well 
with Dipayan Ghosh, a Harvard fellow and former Facebook privacy 
and public policy official who is now with the New America think 
tank. He said the use of advanced algorithms and AI in recruiting 
can create tremendous value for the industry, where discrimination 
by hiring managers has been rampant, but if implemented 
irresponsibly, it can have drastic and harmful effects for job 
candidates. 


"Algorithms discriminate. There have been countless episodes in 
different contexts|tHat have illustrated this in high reķali L> J = 


MARKETS, SINESS NEWS CNBC TV MENU 
recent years, from social media advertising to creditworthiness 





decision-making to subsidy dispensations." 


He also said companies reviewing their own code is not enough, 
especially in the corporate sector, where returns are optimized 
against near-term revenue, forward investment and stock return, 
above all else. "We know of too many past cases where all a 
company needed to do is to self-certify, and it was shown to be 
perpetuating harms to society and, specifically, certain people. ... 
The public will have little knowledge as to whether or not the firm 
really is making biased decisions if it's only the firm itself that has 


https://www.cnbc.com/2018/05/30/silicon-valley-is-stumped-even-a-i-cannot-remove-bias-from-hiring.html 9/14 


8/10/2018 Silicon Valley is stumped, even A.I. cannot remove bias from hiring 
access to its decision-making algorithms to test them for 
discriminatory outcomes." 


"There could be a serious risk, and it has the 
potential to open up the floodgates to 
something very bad." 


-Davida Perry, co-founder and managing partner of Schwartz, Perry & Heller LLP, a firm that 


specializes in employment law 


"The hope is that [using technology in recruiting] will save money 
and take the bias out of the process, but there may be a downside," 
said Davida Perry, co-founder and managing partner of Schwartz, 
Perry & Heller LLP, a firm that specializes in employment law, 
including discrimination cases. "There could be a serious risk, and it 





has the potential tkbpen up the floodgates to somet Er very bad," 


MARKETS BUSINESS NEWS 





Perry said. 


Suppose there is a situation where bias is found in a third party's 
algorithm, leading to discrimination in the hiring process, and a 
corresponding lawsuit follows. Perry said that instead of having one 
applicant with a lawsuit, you would have many, because of how 
much the process has scaled. Companies may try to claim the third 
party as liable for damages, but that may not hold up in court. "If you 
hire a recruiting company and it has biases [in their algorithm], 
you're not going to be able to say, 'I'm so sorry, that's the recruiting 
company's problem.’ If you [as the company] hire them to serve as 


O 
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your agent, I believe that you would be on the hook for damages," 
Perry said. 


Whittaker said algorithm audits need to include experts, advocacy 
groups and academics reviewing them and studying the effects 
they'll have on different populations. "We think that's not happening 
today, and it could lead to serious problems as AI takes off." 


Ghosh said the start-ups in this field don't face enough pressure to 
use outside audit firms: It is not required by law, it costs money and 
would require "tremendous levels" of compliance beyond what 
internal audits likely require. But he does think that recent 
regulation of the technology sector, such as the new European 
privacy directive GDPR, suggests that policy is moving in the right 
direction. He believes algorithm audits are a critical need for the 
public, particularly for people who are and historically have been 
marginalized. 


"Personal prejudices can quickly become reflected in AI," Ghosh 
said. "In recruiting — a space in which sensitive and life-changing 
decisions are made all the time and in which we accordingly have 
established strong civil rights protections — these forms of vicious 
algorithmic bias are especially important to detect and act against." 
— Additional reporting by CNBC Coordinating Producer Krista Braun 
and CNBC news interns Chris Crouse and Rick Morgan 


Talent@Work, the inaugural event of CNBC's @Work 
series, will take place June 20 in NYC. Focusing on the Q 
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workforce of tomorrow, the event will feature IAC 
CEO Joey Levin, LinkedIn head of product @ryros, 
Boxed CEO @Astrochieh and more. REGISTER NOW. 


Eric Rosenbaum 
Editor, CNBC.com 
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THE KILLER ROBOTS ARE HERE: LEGAL AND POLICY 
IMPLICATIONS 


Rebecca Crootof* 


In little over a year, the possibility of a complete ban on autonomous 
weapon systems—known colloquially as “killer robots” —has evolved from a 
proposal in an NGO report to the subject of an international meeting with 
representatives from over eighty states. However, no one has yet put 
forward a coherent definition of autonomy in weapon systems from a law of 
armed conflict perspective, which often results in the conflation of legal, 
ethical, policy, and political arguments. This Article therefore proposes that 
an “autonomous weapon system” be defined as “a weapon system that, 
based on conclusions derived from gathered information and 
preprogrammed constraints, is capable of independently selecting and 
engaging targets.” 

Applying this definition, and contrary to the nearly universal consensus, 
it quickly becomes apparent that autonomous weapon systems are not 
weapons of the future: they exist and have already been integrated into 
states’ armed forces. The fact that such weaponry is currently being used 
with little critique has a number of profound implications. First, it 
undermines pro-ban arguments based on the premise that autonomous 
weapon systems are inherently unlawful. Second, it significantly reduces the 
likelihood that a complete ban would be successful, as states will be 
unwilling to voluntarily relinquish otherwise lawful and uniquely effective 
weaponry. 

But law is not doomed to follow technology: if used proactively, law can 
channel the development and use of autonomous weapon systems. This 
Article concludes that intentional international regulation is needed, now, 
and suggests how such regulation may be designed to incorporate beneficial 
legal limitations and humanitarian protections. 


+ Ph.D. Candidate in Law, Yale Graduate School of Arts and Sciences; Resident Fellow, Yale 
Information Society Project. Thanks to Douglas Bernstein, Eyal Benvenisti, Bonnie Docherty, 
Stuart Ford, Harold Hongju Koh, and Paul Scharre for their thoughtful edits and contributions; to 
Jack Balkin, Lea Brilmayer, Oona Hathaway, Zachary Herz, Margot Kaminski, Ryan Liss, Robert 
Post, Reva Siegel, Julia Spiegel, and my fellow Ph.D.s in Law for their insights and encouragement; 
and to Mark Crootof for my lifelong love of science fiction. This Article also benefited greatly 
from participants’ comments at ASIL’s 2014 Midyear Meeting. 
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INTRODUCTION 


Autonomous weapon systems have long been a bugaboo of science 
fiction. Hal 9000 of 2001: A Space Odyssey, the Cylons of Battlestar 
Galatica, the spider robots of Kabu-Kabu, and the Terminator’s 
namesake all are imagined self-directed lethal robots, terrifying in large 
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part because they do not experience empathy, pity, and mercy—and 
because they might escape all human control.! 

But, contrary to the general consensus,? autonomous weapon 
systems are far from fictional. Weapon systems with varying levels of 
autonomy and lethality have already been integrated into the armed 
forces of numerous states. Israel, Russia, and South Korea reportedly 
have autonomous weapon systems currently patrolling their borders 
and bases; Israel and the United Kingdom have fire-and-forget missiles 
which independently select and engage targets; China and Russia 
employ sea mines which determine when and against what to deploy 
torpedoes; and the United States is developing and using a host of 
autonomous ground, air, and sea-based weapon systems. 

Spurred by dystopic visions of indiscriminate robotic warfare, in 
December 2012, Human Rights Watch and the Harvard International 
Human Rights Clinic issued a report calling for a complete ban on fully 
autonomous weapon systems.3 While not the first such plea, this highly 
publicized report sparked a heated debate on whether such weapons 
should (or could) be banned.4 In April 2013, the conglomerate 
nongovernmental organization Campaign to Stop Killer Robots formed 
for the sole purpose of promoting a ban. One month later, the UN 
Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions 
for the Office of the High Commissioner for Human Rights presented a 


1 To be fair, there are also benign, even invaluable, autonomous weapon systems populating 
science fiction as well, including Lieutenant Commander Data, R2D2, and the reprogrammed 
Terminator of Terminator 2: Judgment Day. 

2 Both proponents and skeptics of a ban tend to agree that, while states may begin integrating 
autonomous weapon systems into their armed forces within the next few decades, such weapons 
do not yet exist. See, e.g., HUMAN RIGHTS WATCH & INTL HUMAN RIGHTS CLINIC, HARVARD 
LAW SCH., LOSING HUMANITY: THE CASE AGAINST KILLER ROBOTS 46 (2012) [hereinafter LOSING 
HUMANITY], available at http://www.hrw.org/sites/default/files/reports/arms1112ForUpload_0_ 
0.pdf (“Although fully autonomous weapons do not exist yet, technology is rapidly moving in that 
direction.”); Michael N. Schmitt & Jeffrey S. Thurnher, “Out of the Loop”: Autonomous Weapon 
Systems and the Law of Armed Conflict, 4 HARV. NATL SECURITY J. 231, 234 (2013) (“[A]n 
outright ban is premature since no such weapons have even left the drawing board.”). 

3 LOSING HUMANITY, supra note 2. 

4 Many join Human Rights Watch and the Harvard International Human Rights Clinic in 
advocating for a complete ban. See, e.g., Mary Ellen O’Connell, Banning Autonomous Killing: The 
Legal and Ethical Requirement that Humans Make Near-Time Lethal Decisions, in THE AMERICAN 
WAY OF BOMBING: CHANGING ETHICAL AND LEGAL NORMS, FROM FLYING FORTRESSES TO 
DRONES 224 (Matthew Evangelista & Henry Shue eds., 2014); Peter Asaro, On Banning 
Autonomous Weapon Systems: Human Rights, Automation, and the Dehumanization of Lethal 
Decision-Making, 94 INTL REV. RED CROSS 687 (2012); Noel E. Sharkey, The Evitability of 
Autonomous Robot Warfare, 94 INT'L REV. RED CROSS 787 (2012). 

There are a number of vocal skeptics of such a ban. See, e.g., Shane R. Reeves & William J. 
Johnson, Autonomous Weapons: Are You Sure These Are Killer Robots? Can We Talk About It?, 
2014 ARMY LAW. 25, 31 (2014); Schmitt & Thurnher, supra note 2, at 234; Kenneth Anderson & 
Matthew Waxman, Law and Ethics for Autonomous Weapon Systems: Why a Ban Won't Work 
and How the Laws of War Can, HOOVER INST. (Apr. 9, 2013), http://media.hoover.org/sites/ 
default/files/documents/Anderson-Waxman_LawAndEthics_r2_FINAL.pdf. 
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report to the Human Rights Council questioning whether the 
deployment of autonomous weapon systems was permissible.s Many 
state delegates suggested that the topic be addressed at the upcoming 
meeting of state parties to the Convention on Certain Conventional 
Weapons—and, in November, the state parties to the Convention 
agreed. Consequently, in May 2014, representatives from over eighty 
states and from United Nations agencies, civil society, and other 
international and transnational organizations attended a “Meeting of 
Experts on Lethal Autonomous Weapons Systems” to discuss the 
possibility of a complete ban on such weaponry.6 

Ban proponents have good reason to be enthused: this may well be 
one of the swiftest campaigns in history to ban a class of weaponry.” But 
in the excitement of the progressing debate, neither side has managed to 
construct a coherent definition for autonomous weapon systems for the 
purpose of a weapons ban. Instead, “autonomy” in weapon systems 
often means different things to different stakeholders, and as a result, 
state representatives, developers, military lawyers, human rights 
activists, philosophers, and other policymakers often talk past each 
other.’ Indeed, during the recent international Experts Meeting on 
Autonomous Weapon Systems, multiple states noted the need for 
clarification. Thus, not only is a definition of autonomy for weapon 
systems from a law of armed conflict perspective necessary to fill a gap 
in the legal literature, it is crucial to current and ongoing treaty 
discussions. 


5 Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, Report of the 
Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, 20-21, Human Rights 
Council, U.N. Doc. A/HRC/23/47 (Apr. 9, 2013) [hereinafter Heyns Report] (written by Christof 
Heyns and calling for national moratoria on the testing, production, assembly, transfer, 
acquisition, deployment, and use of lethal autonomous weapons). 

6 Jean-Hugues Simon-Michel, Report of the 2014 Informal Meeting of Experts on Lethal 
Autonomous Weapons Systems (LAWS), UN OFF. GENEVA (May 16, 2014), http://www.unog.ch/ 
80256EDD006B8954/%28httpAssets%29/350D9ABED1AFA515C1257CF30047A8C7/$file/ 
Report_AdvancedVersion_10June.pdf [hereinafter Chairperson Simon-Michel Report]. A follow- 
up meeting was held in April 2015. 

7 See Sarah Knuckey, Start of First Inter-Governmental Expert Meeting on Autonomous 
Weapons, JUST SECURITY (May 13, 2014, 3:17 PM), http://justsecurity.org/2014/05/13/start-inter- 
governmental-expert-meeting-autonomous-weapons (quoting the Netherlands representative as 
saying “never before has a disarmament issue gained interest so quickly”). 

8 See, e.g., ARMIN KRISHNAN, KILLER ROBOTS: LEGALITY AND ETHICALITY OF AUTONOMOUS 
WEAPONS 43 (2009) (describing political, philosophical, and technical definitions of “autonomy” 
and observing that, “[a]s the discourse on autonomous robots gets seized more and more by 
philosophers ...the confusion about ‘autonomous weapons’ in the public debate increases”); 
Kathleen Lawand, Fully Autonomous Weapon Systems, INT'L COMMITTEE RED CROSS (Nov. 25, 
2013), http://www.icrc.org/eng/resources/documents/statement/2013/09-03-autonomous- 
weapons.htm (noting that, although there is “a wealth [of] expert literature on this subject, there 
is somewhat of a lack of consistency in the use of terms”). 

9 See Knuckey, supra note 7. 
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Part I begins with a review of existing definitions and their flaws. It 
then proposes a new definition for autonomy in weapon systems, which 
turns on the central issue in legal debates regarding a weapon ban— 
whether a human being must take an affirmative action before lethal 
force is used against a specific target. Accordingly, an “autonomous 
weapon system” is “a weapon system that, based on conclusions derived 
from gathered information and preprogrammed constraints, is capable 
of independently selecting and engaging targets.” With this clarified 
definition it quickly becomes apparent that autonomous weapon 
systems are not weapons of the future: they exist and are in use today. 
This is not to say there is not ample space along the autonomy spectrum 
from weaponry currently in use to the Terminator—but that space is far 
less significant than the distinction between a weapon that cannot use 
lethal force against a specific target without a human operator’s 
affirmative action and one that can. 

The fact that autonomous weapon systems are already in use has 
two profound implications. First, as discussed in Part II, it undermines 
arguments that autonomous weapon systems are inherently illegal. Ban 
advocates make a number of important moral, policy, and strategic 
arguments, but their primary legal claim is that autonomous weapon 
systems will never be able to comply with the law of armed conflict. 
Specifically, they argue that autonomous weapon systems will not be 
able to distinguish between lawful and unlawful targets; that they will 
not be able to conduct in bello proportionality assessments, which entail 
determining whether the military objective to be gained is worth the risk 
of likely collateral damage; and that their use may not accord with the 
Martens Clause of the First Additional Protocol to the 1949 Geneva 
Conventions—which, ban proponents argue, requires new technology 
to comply with “the principles of humanity” and “the dictates of the 
public conscience.” Some question whether states will be able to hold 
individuals accountable for war crimes committed by autonomous 
weapon systems; still others posit that the decision to kill another 
human being can never be lawfully delegated to a machine. Ban skeptics, 
responding to these arguments, dispute the associated analyses—but 
they accept the proponents’ assumption that autonomous weapon 
systems are futuristic weaponry. However, insofar as such weaponry is 
currently being used with little to no critique, legal arguments for a ban 
lose their force. Autonomous weapon systems now in use are being 
lawfully employed—thus, such weapons as a class are not inherently 
unlawful. 

Should autonomous weapon systems nonetheless be banned? Some 
ban proponents are willing to concede that these weapons are not per se 
unlawful, but argue that they should nonetheless be banned for a host of 
other reasons. However, the fact that autonomous weapon systems are 
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already in use significantly reduces the likelihood that states will enact 
an effective ban. Scholars on both sides of the ban debate have mined 
history for evidence that states will or will not be able to successfully ban 
autonomous weapon systems, but by and large they provide little 
analysis of which of these precedents are most apt. Part III therefore 
discusses eight qualities which seem to increase the likelihood that a 
given ban will be successful: the weapon causes superfluous injury or 
unnecessary suffering; the weapon is inherently indiscriminate; the 
weapon is ineffective; other means exist for accomplishing a similar 
military objective; the ban is clear and narrowly tailored; there has been 
related prior regulation; there is significant public concern and civil 
society engagement; and there is sufficient state commitment. Of these, 
only one characteristic—civil society engagement—suggests that a ban 
on autonomous weapon systems would be successful; the others are 
either inconclusive or currently weigh against the likelihood of a 
successful ban. 

“Killer robots” are here, and they are here to stay. But law is not 
doomed to follow technology: if used proactively, law can channel the 
development and use of autonomous weapon systems. Part IV therefore 
considers the question of how such weaponry might be effectively 
regulated. After discussing the need for intentional international 
lawmaking, now, this Part concludes with concrete suggestions as to 
how it might be designed to incorporate many of the beneficial legal 
limitations and humanitarian protections associated with a complete 
ban. 


I. WHAT’S IN A NAME? 


Autonomous Weapon Systems. Killer Robots. Unmanned Lethal 
Weapons. Lethal Autonomous Robots. Many writers on this subject use 
the terms “autonomous,” “robots,” “lethal,” and “weapon systems” in 
various combinations, often presuming that the reader will intuitively 
grasp the distinction between hypothetical, futuristic weaponry and 
existing, lawful weapon systems. 

But as there is no coherent definition of “autonomy” in weapon 
systems, that distinction is far from clear. Why, for example, do some 
consider a weapon system that independently identifies, tracks, and 
engages an incoming threat merely “automated” or “automatic,” but rail 
against prospective “autonomous” weapon systems that would 
independently identify, track, and engage a target? Until there is a 
shared legal definition of autonomy for weapon systems, it will be 
impossible to have a productive conversation about what a new treaty 
should ban or regulate. 
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After reviewing existing definitions and their flaws, this Part 
proposes a new definition for autonomy in weapon systems. It should be 
acknowledged at the outset that this definition, like any definition in 
law, is not an independent, abstract formulation. Instead, its 
construction is necessarily tied to the purpose it is designed to serve— 
clarifying what should constitute an autonomous weapon system from a 
law of armed conflict perspective. As a result, this definition turns on 
whether human involvement is necessary for a weapon system to 
exercise lethal force against a specific target. Different legal regimes or 
different disciplines might require definitions that emphasize other 
attributes of autonomous weapon systems. 

This new definition allows for a relatively clear categorization of 
existing weapon systems into inert, automated, semi-autonomous, and 
autonomous weapon systems.!0 Once these categories are delineated, a 
surprising fact quickly becomes clear: contrary to the nearly universal 
consensus, autonomous weapon systems currently exist and have 
already been integrated into states’ armed forces. As discussed in greater 
detail in the remainder of this Article, this fact has profound 
implications for the discussion over how such weapons should be 
regulated. 


A. The Need for a Law of Armed Conflict-Based Definition 


“Autonomy” carries vastly different meanings in different fields. A 
political scientist might define autonomy as the ability to be self- 
governing; a philosopher might focus on an _ entity’s moral 
independence; an engineer might be concerned with a machine’s level of 
dependence on human beings in completing different tasks. Due in part 
to these differing understandings of autonomy, various stakeholders in 
the debate over banning autonomous weapon systems often speak past 
each other. And, as discussions of autonomy in weapon systems are 
“fraught with terms that are both loaded and vague,”!! they can easily 
become heated and unproductive—especially as none of the 
aforementioned definitions provide an appropriate orientation for 


10 The term “semi-autonomous” is somewhat misleading, as it implies that autonomy in 
weapon systems is not a binary characteristic. However, in accordance with other writing in the 
field, this Article uses it to describe weapon systems that have some autonomous capabilities but 
which cannot independently select and engage targets. A drone that suggests a target to a human 
operator, but which cannot engage that target without approval, would be semi-autonomous; a 
drone which could select and engage targets post-deployment without human involvement would 
be autonomous. As discussed below, this distinction is complicated by the fact that otherwise 
autonomous weapon systems may be operated in semi-autonomous modes. 

11 William C. Marra & Sonia K. McNeil, Understanding “The Loop”: Regulating the Next 
Generation of War Machines, 36 HARV. J.L. & PUB. POL’Y 1139, 1143 (2013). 
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defining “autonomy” for weapon systems in the context of treaty 
negotiations. A treaty banning or regulating a class of weaponry is 
fundamentally a legal instrument memorializing an agreement between 
states.12 To draft a treaty regarding the creation, use, or transfer of 
autonomous weapon systems, it is necessary to determine what 
“autonomy” in weapon systems entails from a law of armed conflict 
perspective. 

While developments in technology regularly challenge basic 
precepts of law, autonomous weapon systems threaten one of the law of 
armed conflicts most fundamental assumptions: that, ultimately, a 
human being decides whether another human being lives or dies. 
Various automated weapon systems, like anti-personnel landmines, 
have tested this principle in the past, but the prospect of a fully 
autonomous weapon system strikes a more visceral note. As former U.S. 
Major General Robert Latiff describes it, “[flull lethal autonomy is no 
mere next step in military strategy: [i]t will be the crossing of a moral 
Rubicon.”!3 No longer will responsibility for the consequences of a 
decision to use lethal force be directly traceable to a human operator; 
instead, responsibility may rest with the operator, the military 
commander, the programmer, the manufacturer, the weapon system 
itself, or some combination thereof. Thus, part of the purpose of a 
definition for autonomy in weapon systems from a law of armed conflict 
perspective is to distinguish between traditional weaponry and 
associated responsibility regimes and this new form.!4 Accordingly, the 
distinctions between a non-autonomous, semi-autonomous, and 
autonomous weapon systems should turn on the level of human 
involvement necessary for the system to exercise lethal force against a 
specific target. 

In attempting to evaluate levels of human control, it is tempting to 
rely on an engineer’s definition of robotic autonomy. But engineers do 
not measure robotic autonomy along a single continuum; instead, 
machines are understood as progressively more autonomous based on 
various qualities, including how frequently the robot must be in contact 
with a human operator, how well it functions in response to increasing 
levels of uncertainty in its environment, and its “assertiveness”—its 
ability to alter its operating plan to complete its mission.15 Evaluating 


12 Vienna Convention on the Law of Treaties art. 2, May 23, 1969, 1155 U.N.T\S. 331. 

13 Robert H. Latiff & Patrick J. McCloskey, Opinion, With Drone Warfare, America 
Approaches the Robo-Rubicon, WALL ST. J. (Mar. 14, 2013, 7:37 PM), http://online.wsj.com/news/ 
articles/SB10001424127887324128504578346333246145590. 

14 See Asaro, supra note 4, at 695 (“It is the delegation of the human decision-making 
responsibilities to an autonomous system designed to take human lives that is the central moral 
and legal issue.”). 

15 Marra & McNeil, supra note 11, at 1151-55; see also ANDREW P. WILLIAMS, 
MULTINATIONAL CAPABILITY DEVELOPMENT CAMPAIGN (MCDC), TYPOLOGICAL ANALYSIS: 
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autonomy along these different axes introduces a confusing and 
unnecessary level of particularity. Not only might a weapon system have 
varying levels of autonomy with regard to each of these three attributes, 
it may have differing amounts of autonomy with regard to each of these 
qualities at each stage of the “OODA Loop” (a simplified description of 
human decisionmaking as a four-step process: Observe, Orient, Decide, 
Act).!6 One weapon system might be extremely assertive at the Observe 
and Orient stages, but not at the Decision and Action stages; another 
might have a high degree of independence at the Action stage, but not at 
the Observe, Orient, or Decide stages. These varying levels of autonomy 
along multiple cognitive axes at the four stages of the OODA Loop 
result in complicated and difficult-to-compare technical distinctions. 

Although these gradations may be extremely useful in research and 
development,!’ they are unnecessarily precise for a legal document that 
is ultimately concerned with regulating weaponry that might 
independently exercise lethal force. In fact, a purely mechanical 
definition of autonomy for weapon systems might have the perverse 
effect of distinguishing among different weapon systems for technical 
reasons regardless of whether they operate similarly in practice, or vice 
versa. 

Instead, a relevant definition of autonomy for weapon systems 
should turn on whether a human being’s affirmative action is necessary 
for a weapon system to use lethal force against a specific target. The 
definition ultimately proposed in this Article therefore conceives of 
autonomy in weapon systems as both a binary characteristic and as 
existing along as spectrum. A given weapon system will or will not be 
autonomous,!§ but autonomous weapon systems may have differing 
degrees of autonomous capabilities. 


AUTONOMOUS SYSTEMS FOCUS AREA 18-28 (2014) (collecting and discussing problems with 
various scales of machine autonomy); Paul Scharre & Michael C. Horowitz, An Introduction to 
Autonomy in Weapon Systems 5-7 (Feb. 2015) (working paper), available at 
http://www.cnas.org/sites/default/files/publications-pdf/Ethical%20Autonomy%20Working%20 
Paper_021015_v02.pdf (discussing three independent dimensions of weapons autonomy: the 
human-machine command-and-control relationship, the machine’s complexity, and the type of 
function being automated). 

16 See Marra & McNeil, supra note 11, at 1144-45; see also id. at 1146-49 (comparing how a 
human being and machine might make a decision based on the OODA loop framework). 

17 For example, the Air Force Research Lab employs an eleven-level spectrum of robotic 
autonomy, which requires an analysis of how much independence a given system has with regard 
to the four OODA tasks to determine its overall level of autonomy. See id. at 1157-58. 

18 See Mark Gubrud, Autonomy Without Mystery: Where Do You Draw the Line?, 1.0 HUM. 
(May 9, 2014), http://gubrud.net/?p=272 (arguing for a definition of “human control” that is “free 
of degrees of meaning”). 
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B. Existing Definitions and Their Problems 


The U.S. Department of Defense’s (DoD) definition, which was 
made public in a 2013 Directive, is currently the best and most 
commonly cited definition for autonomous weapon systems. While the 
definition standing alone is useful, the Directive’s ambiguities invite 
certain misreadings and fail to logically distinguish between 
autonomous and semi-autonomous weapon systems. 

Few legal scholars have advanced alternative definitions. To the 
extent other definitions have been stated or may be inferred, they tend 
to be either overly inclusive or exclusive. Some set the bar for autonomy 
in weapon systems too low, eliding important distinctions between 
different levels of human involvement. Others set the bar too high, 
effectively defining autonomous weapon systems out of existence and 
thereby ignoring a host of issues associated with weapon systems in use 
today. Finally, some definitions propose distinctions based on factors 
which might be highly relevant in other fields, such as engineering or 
philosophy, but which are less appropriate when attempting to evaluate 
relative levels of human control over target selection and engagement. 


1. The DoD’s Definition 


In its 2013 Directive, the DoD defined “autonomous weapon 
systems” as ones which, “once activated, can select and engage targets 
without further intervention by a human operator. This includes 
human-supervised autonomous weapon systems that are designed to 
allow human operators to override operation of the weapon system, but 
can select and engage targets without further human input after 
activation.” 19 

This definition has a number of advantages. By beginning with 
“once activated,” it highlights how human beings are initially 
responsible for the decision to deploy autonomous weapon systems. It 
underscores the fact that it is the delegation of target selection and 
engagement (as opposed to other functions or tasks, such as piloting) to 
a machine that makes a weapon system an “autonomous weapon 
system.” It also does not create an unnecessary distinction between 
“autonomous” and “fully autonomous” weapon systems:2° rather, a 
system’s autonomy is determined solely by whether it “can select and 
engage targets without further intervention by a human operator.” 


19 U.S. DEPT OF DEF., DIRECTIVE NO. 3000.09: AUTONOMY IN WEAPON SYSTEMS 13-14 
(2012) [hereinafter DOD DIR. 3000.09]. 
20 See infra Part I.C.4.c. 
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Finally, it spells out that autonomy for weapon systems depends on the 
system’s capability for autonomous action—not whether it is supervised 
in practice or used in that capacity. 

Unfortunately, this apparently clear definition is muddied by the 
Directive’s attempt to distinguish between “autonomous” and “semi- 
autonomous” weapon systems. It defines the latter as those which “once 
activated, [are] intended to only engage individual targets or specific 
target groups that have been selected by a human operator.”2! At first, 
this distinction seems appropriate: autonomous weapon systems “can 
select and engage targets without further intervention,” but semi- 
autonomous weapon systems may “only engage [targets] selected by a 
human operator.” But the crucial distinguishing factor between 
autonomous and semi-autonomous weapon systems in these 
definitions—human responsibility for target selection—is vague, and 
the provided examples of semi-autonomous weapon systems confuse 
rather than clarify.” 

According to the Directive, semi-autonomous weapon systems 
include those “that employ autonomy for engagement-related 
functions ... provided that human control is retained over the decision 
to select individual targets and specific target groups for engagement” 
and “‘[flire and forget’ or lock-on-after-launch homing munitions that 
rely on [tactics, techniques, and procedures] to maximize the probability 
that the only targets within the seeker’s acquisition basket when the 
seeker activates are those individual targets or specific target groups that 
have been selected by a human operator.”23 Presumably, targets 
preselected by human operators would need to be fairly specific, limited, 
or predictable to preserve a meaningful distinction between semi- 
autonomous and autonomous weapon systems.”4 

Guided munitions that “lock on” prior to launch easily meet the 
specificity requirement. Those which “lock on” afterwards are more 
problematic. The Directive defines as “semi-autonomous” any homing 
munition that, after deployment, independently identifies targets and 
engages them based on tactics, techniques, and procedures proscribed 
by a human operator.25 The DoD does not consider this autonomous 
“target selection,” insofar as constraints on what the munition may 
target are determined by a human being and its infrastructure. But this 
blurs the line between autonomous and semi-autonomous weapon 
systems, as autonomous weapon systems will also operate under an 


21 DOD DIR. 3000.09, supra note 19, at 14. 

22 See Gubrud, supra note 18 (discussing ambiguities in the DoD’s definition of target 
selection). 

23 DOD DIR. 3000.09, supra note 19, at 14. 

24 See infra Part I.C.4.a. 

25 DOD DIR. 3000.09, supra note 19, at 14. 
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array of preprogrammed and practical constraints; an autonomous 
weapon system will not simply be directed to “eliminate the enemy.” 
Some have therefore concluded that such “lock-on-after-launch homing 
munitions” actually are making lethal decisions autonomously—and, as 
the Directive “places no upper limit on the sophistication of the sensors 
and computers or complexity of the algorithms,” nor does it limit 
“homing munitions” to flying objects, “this is a loophole The 
Terminator could walk through.” 2s 

Additionally, the Directive’s definition of semi-autonomous 
weapon systems sometimes privileges form over function. In one case, 
“human control...over the decision to select individual targets and 
specific target groups for engagement” could constitute a human 
operator identifying and selecting targets, while the weapon system 
merely exercises autonomy in “providing terminal guidance to home in 
on” them.?’ In another, “human control” might consist only of a human 
being not vetoing the engagement of a target the weapon system had 
acquired, tracked, identified, and prioritized—and which possibly poses 
an incoming threat which must be neutralized as quickly as possible, 
leaving little time for considered evaluation of the situation.28 In the 
latter case, the human being only nominally exercises control, insofar as 
he tacitly approves an engagement—the weapon system is effectively 
using lethal force with no genuine human supervision or involvement.?9 

Shortly before this Article was finalized for publication, the Center 
for New American Security’s Ethical Autonomy project advanced a new 
definition intended to address some of these concerns.30 It defines an 
autonomous weapon system as “a weapon system that, once activated, is 
intended to select and engage targets where a human has not decided 
those specific targets are to be engaged.”3! It distinguishes these from 
“human-supervised autonomous weapon systems,” which have “the 
characteristics of an autonomous weapon system, but with the ability for 
human operators to monitor the weapon system’s performance and 
intervene to halt its operation, if necessary.”32 It also distinguishes a 
“semi-autonomous weapon” as one which “incorporates autonomy into 
one or more targeting functions and, once activated, is intended to only 
engage individual targets or specific groups of target[s] that a human 
has decided are to be engaged.” 33 


26 Gubrud, supra note 18. But see Scharre & Horowitz, supra note 15, at 8-10 (arguing that 
both types of guided munitions engage only targets preselected by a human operator). 

27 DOD DIR. 3000.09, supra note 19, at 14. 

28 Id. 

29 See infra Part I.C.4.b-c. 

30 Scharre & Horowitz, supra note 15, at 16. 

31 Id. 

32 Id. 

33 Id. 
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These definitions share some of the positive qualities of the 
Directive’s definition, in that they highlight the importance of target 
selection and engagement and the human decision to activate the 
weapon system. They also improve upon the Directive’s definition by 
clarifying the role of human decisionmaking and the importance of the 
specificity of the preselected target. 

Unfortunately, these definitions create a different type of 
confusion. The distinction between “autonomous weapon systems” and 
“human-supervised autonomous weapon systems” is phrased as a 
technical one, insofar as the latter permit human monitoring and 
intervention. But this raises two issues. First, because nothing in the 
definition for “human-supervised” systems requires such monitoring 
and intervention, this distinction might be rendered moot if, in practice, 
such systems are operated without supervision. Second, despite being 
cast as technical, this distinction is ultimately a question of possible 
usage or modes, not autonomy. As such, it may well be important— 
even crucial—in creating research and design regulations, but not in 
differentiating between autonomous and non- or semi-autonomous 
systems.34 


2. Irrelevant Distinctions 


There are a number of definitions cited in the literature on banning 
autonomous weapon systems that introduce distinctions between “non- 
autonomous” and “autonomous” weapon systems which, while likely 
useful in certain contexts, are irrelevant when constructing a regulatory 
treaty’s definition—usually because they fail to account for the level of 
human involvement in the decision to use lethal force. 

For example, the DoD’s Roadmap appears to define “automatic” 
unmanned systems as those which cannot “initially define the path 
according to some given goal or to choose the goal that is dictating its 
path.”35 An “autonomous system,” in contrast, “is self-directed by 
choosing the behavior it follows to reach a human-directed goal.”3 
Putting aside the fact that all robotic systems will be extensively 
preprogrammed and the difficulty in determining when a system’s 
controlling algorithms become sufficiently complex to state that it is 
“choosing” its actions, the Roadmap’s distinction turns on the level of 
independence the system has in accomplishing a goal—which has 
nothing to do with what the goal is. Should the goal require selecting 


34 See infra Part I.C.4.c. 

35 U.S. DEPT OF DEF., UNMANNED SYSTEMS INTEGRATED ROADMAP FY 2013-2038, at 66 
(2013) [hereinafter DOD ROADMAP]. 

36 Id. at 67. 
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and engaging targets, the degree of human involvement is highly 
relevant. But weapon systems need not have destructive goals: one 
useful weapon system might be an autonomous version of the U.S. 
Miniature Air Launched Decoy Jammer, an air vehicle designed to 
“confuse and deceive enemy [integrated air defense systems]” by 
cruising in enemy territory, emitting decoy signals, and jamming the 
electromagnetic spectrum.” 

Similarly, Noel Sharkey, a Professor of Artificial Intelligence and 
Robotics at the University of Sheffield and a current Chair of the 
International Committee for Robot Arms Control, defines an 
“automatic” robot as one which “carries out a pre-programmed 
sequence of operations or moves in a structured environment.”38 In 
contrast, he defines an “autonomous” robot as “similar to an automatic 
machine except that it operates in open or unstructured environments. 
The robot is still controlled by a program but now receives information 
from its sensors that enable it to adjust the speed and direction of its 
motors (and actuators) as specified by the program.”39 According to this 
distinction, the question of a weapon system’s autonomy might depend 
on the environment in which it is employed—and not on its task or, 
should the task involve lethal force, on the level of human involvement. 
A stationary weapon system operating in a constrained area would be 
automatic—but that same weapon operating in an unstructured 
environment would be autonomous. But what constitutes a “structured” 
environment in warfare? Additionally, to the extent this definition 
depends on where a weapon system is deployed, it would make the 
enforcement of a ban on “autonomous” weaponry impossible. Despite 
this irrelevant distinction, Sharkey’s definition has pervaded discussions 
regarding what weapon systems should be included in a ban. The 
authors of Losing Humanity, for example, cite Sharkey for their claim 
that many weapon systems in use today with “a significant degree of 
autonomy because they can sense and attack targets with minimal 
human input” nonetheless are “better classified as automatic.” 40 


3. Setting the Bar for Autonomy Too Low 


There are a few proffered definitions that set the bar for autonomy 
so low that they lump together wide varieties of existing and potential 
weapons, and therefore “almost certainly [miss] the essence of what is 


37 Miniature Air Launched Decoy (MALD), RAYTHEON, §http://www.raytheon.com/ 
capabilities/products/mald (last visited Apr. 27, 2015). 

38 Noel Sharkey, Comment, Automating Warfare: Lessons Learned from the Drones, 21 J.L. 
INFO. & SCI. 140, 141 (2011). 

39 Id. 

40 LOSING HUMANITY, supra note 2, at 12. 
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new about autonomous weapons.”4! Under these definitions, the most 
rudimentary landmine and Hal 9000 from 2001: A Space Odyssey are 
equivalently “autonomous.” 

Peter Asaro, a Professor of Philosophy who studies the 
implications of military robotics, defines autonomous weapon systems 
as “any automated system that can initiate lethal force without the 
specific, conscious, and deliberate decision of a human operator, 
controller, or supervisor.”4 Admirably, Asaro is concerned with 
ensuring meaningful human control over life-and-death decisions. But 
his definition elides important distinctions between weapon systems 
with vastly different levels of human involvement in the decision to use 
lethal force. 

Mark Gubrud, a member of the International Committee on Robot 
Arms Control, argues that “[w]e should...seek principles and 
definitions that point, as directly as possible to a Yes or No answer to 
the question, ‘Is this an autonomous weapon?’”43 Given the need for a 
clear definition for autonomy in weapon systems, Gubrud is asking the 
right question. But his solution—“A system is autonomous if it is 
operating without further human intervention” 44—is incomplete. Even 
assuming that he meant to limit his definition of autonomy to systems 
capable of using lethal force, Gubrud’s definition includes weapon 
systems with vastly disparate levels of human involvement. 


4. Setting the Bar for Autonomy Too High 


Notwithstanding the DoD’s calm pronouncement that “[h]uman- 
supervised autonomous weapon systems may be used to select and 
engage targets,”45 most legal scholars assume that autonomous weapon 
systems do not yet exist.4¢ This conclusion is usually grounded in 
alternative definitions of autonomy for weapon systems, which tend to 
have two complimentary issues: an overly broad definition of non- 
autonomous (usually termed “automated” or “automatic”) weapon 
systems, and an overly narrow definition of autonomous weapon 


41 Paul Scharre, Autonomy, “Killer Robots,” and Human Control in the Use of Force—Part I, 
JUST SECURITY (July 9, 2014, 11:17 AM), http://justsecurity.org/12708/autonomy-killer-robots- 
human-control-force-part [hereinafter Scharre, Autonomy I]; cf. Michael C. Horowitz & Paul 
Scharre, Do Killer Robots Save Lives?, POLITICO (Nov. 19, 2014), http://www.politico.com/ 
magazine/story/2014/1 1/killer-robots-save-lives-113010_full.html?print#. VICWzWds2dx 
(discussing concerns that ban advocates are setting the bar for “killer robots” inappropriately low 
to encompass life-saving precision-guided munitions). 

42 Asaro, supra note 4, at 694. 

43 Gubrud, supra note 18. 

44 Id. 

45 DOD DIR. 3000.09, supra note 19, at 3. 

46 See supra note 2. 
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systems. Such definitions are problematic, however, insofar as they 
ignore the fact that weapon systems in use today can make independent 
determinations regarding the selection and engagement of targets. 

The U.K. Ministry of Defence defines “autonomous systems” as 


capable of understanding higher level intent and direction. From this 
understanding and its perception of its environment, such a system is 
able to take appropriate action to bring about a desired state. It is 
capable of deciding a course of action, from a number of alternatives, 
without depending on human oversight and control, although these 
may still be present. Although the overall activity of an autonomous 
unmanned aircraft will be predictable, individual actions may not 
be.47 


It continues: 


Autonomous systems will, in effect, be self-aware and their response 
to inputs indistinguishable from, or even superior to, that of a 
manned aircraft. As such, they must be capable of achieving the same 
level of situational understanding as a human. ... As computing and 
sensor capability increases, it is likely that many systems, using very 
complex sets of control rules, will appear and be described as 
autonomous systems, but as long as it can be shown that the system 
logically follows a set of rules or instructions and is not capable of 
human levels of situational understanding, then they should only be 
considered to be automated.48 


This position exemplifies the problems inherent in setting the bar 
for weapon systems’ autonomy too high. Robots are unlikely to achieve 
“the same level of situational understanding as a human”* any time 
soon, rendering any policies based on this definition largely 
hypothetical. A weapon system that independently selects and engages 
targets based on gathered data or even on in-field learning would be 
classified as merely “automated” as long as it didn’t have human-level 
cognitive capabilities. Thus, although the United Kingdom has publicly 
stated a policy against employing autonomous weapon systems, it has 
minimal practical impact or import.5° 


47 U.K. MINISTRY OF DEFENCE, JOINT DOCTRINE NOTE 2/11: THE UK APPROACH TO 
UNMANNED AIRCRAFT SYSTEMS 2-3 (2011). 

48 Id. at 2-3 to 2-4. 

49 Id. at 2-3. 

50 See Sharkey, supra note 38, at 141 (critiquing this definition on the grounds that “no system 
is capable of ‘understanding’ never mind ‘understanding higher level intent”); see also NICHOLAS 
MARSH, PEACE RESEARCH INST. OSLO, DEFINING THE SCOPE OF AUTONOMY: ISSUES FOR THE 
CAMPAIGN TO STOP KILLER ROBOTS 2 (2014) (discussing Sharkey’s critique). 
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C. A Clarified Definition 


Given the ambiguities and problems attendant upon other 
definitions, this section suggests a clarified definition of “autonomous 
weapon system,” meant to highlight the unique issues such weaponry 
poses from a law of armed conflict perspective: 


An “autonomous weapon system” is a weapon system that, based on 
conclusions derived from gathered information and preprogrammed 
constraints, is capable of independently selecting and engaging targets. 


The remainder of this section elaborates upon different aspects of this 
definition. 


1. “An ‘autonomous weapon system’ is a weapon system that, ...” 


A “weapon system” is “[a] combination of one or more weapons 
with all related equipment, materials, services, personnel, and means of 
delivery and deployment (if applicable) required for self-sufficiency.”5! 
For autonomous weapon systems, this may include varying 
combinations of physical mechanisms and nonphysical code or 
software. 

A weapon system need not include lethal weapons; certain 
weapons are “explicitly designed and primarily employed so as to 
incapacitate personnel or materiel, while minimizing fatalities, 
permanent injury to personnel, and undesired damage to property and 
the environment.”52 

It also bears noting that weapon systems are not necessarily 
embodied entities: computer viruses, worms, and other malware may 
also operate automatically or autonomously. Although much about 
Stuxnet, a computer worm discovered in early 2010, is shrouded in 
secrecy, it was apparently designed to attack industrial Programmable 
Logic Controllers (PLCs)—specifically, Iranian PLCs controlling 
centrifuges used for enriching weapons-grade uranium.% At some point 
between late 2009 and early 2010, Iran replaced approximately 1000 
centrifuges at its Natanz plant, which has been widely attributed to 
damage caused by Stuxnet.54 Whether Stuxnet succeeded in its mission 
is unknown: 


51 Weapon System, DOD DICTIONARY MIL. TERMS, http://www.dtic.mil/doctrine/dod_ 
dictionary/data/w/7965.html (last visited Apr. 27, 2015). 

52 Nonlethal Weapon, DOD DICTIONARY MIL. TERMS, http://www.dtic.mil/doctrine/dod_ 
dictionary/data/n/11245.html (last visited Apr. 27, 2015). 

53 Stuxnet was likely developed by the United States and Israel. David E. Sanger, Obama 
Order Sped up Wave of Cyberattacks Against Iran, N.Y. TIMES, June 1, 2012, at Al. 

54 DAVID ALBRIGHT, PAUL BRANNAN & CHRISTINA WALROND, INST. FOR SCI. & INT’L SEC., 
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If Stuxnet’s goal was the destruction of all the centrifuges in the [Fuel 
Enrichment Plants], Stuxnet failed. But if its goal was to destroy a 
more limited number of centrifuges and set back Iran’s progress in 
operating [Fuel Enrichment Plants] while making detection of the 
malware difficult, it may have succeeded, at least for a while.55 


But one thing is clear: Stuxnet was a code-based weapon system, albeit 
one without lethal effect. 5s 


2. “... based on conclusions derived from gathered information and 
preprogrammed constraints, ...” 


This clause attempts to distinguish between “automated” and 
“autonomous” weapon systems. Both may gather information, both may 
operate under preprogrammed constraints, and both may engage targets 
independent of human intervention. But while automated weapon 
systems merely react to triggers, autonomous weapon systems process 
information to derive conclusions before responding.5 For example, a 
typical landmine is an automated weapon that uses gathered 
information—such as a tug on a tripwire or pressure on a sensor—to 
trigger an explosion without human involvement or oversight. A 
landmine with autonomous capabilities, however, might be triggered to 
react by a similar tug or pressure, but it would then use algorithms to 
process data (possibly to determine whether or not the trigger was due 
to a child or a tank) and, based on its calculations, reach a conclusion 
about whether or not to explode. More advanced weapon systems with 
autonomous capabilities might even make probabilistic calculations, 
deploy different graduated outcomes based on environmental factors, or 
learn from prior experiences. 

This clause might be criticized—with some justification—as 
attempting to draw an arbitrary line in the sand. From a certain 
perspective, a landmine that is triggered and then explodes seems to be 
just a simplistic version of a landmine that evaluates the weight of a 
trigger before exploding—both operate after activation without further 
human intervention. Any attempt to distinguish between autonomous 
and non-autonomous weapon systems based on their complexity will 


DID STUXNET TAKE OUT 1,000 CENTRIFUGES AT THE NATANZ ENRICHMENT PLANT? 1 (2010). 

55 Id. 

56 See Oona A. Hathaway, Rebecca Crootof, Philip Levitz, Haley Nix, Aileen Nowlan, William 
Perdue & Julia Spiegel, The Law of Cyber-Attack, 100 CALIF. L. REV. 817, 839-40 (2012) 
(discussing various types of cyber-attacks, including Stuxnet). 

57 In other words, autonomous weapon systems “select among” potential targets; automated 
ones are simply triggered. Kenneth Anderson, Daniel Reisner & Matthew Waxman, Adapting the 
Law of Armed Conflict to Autonomous Weapon Systems, 90 INT'L L. STUD. 386, 388 (2014). 
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run into a similar line-drawing problem, which might argue for striking 
this clause entirely.58 

That concern being acknowledged, the fact that a weapon system is 
capable of at least a minimal level of independent analysis does seem to 
be a relevant distinction. This clause highlights that an autonomous 
weapon systems, like any robotic or code-based system, is controlled by 
a program.5 These programs may be created by human beings or by 
other programs, and their constraints may include anything from the 
law of armed conflict to specific parameters of a given mission. Even 
weapon systems with in-field machine learning capabilities will be 
constrained in what they can learn and do by their programs. Thus, 
even though autonomous weapon systems’ responses to certain 
environments or events may be largely predictable—just as the actions 
of an autonomous human being may be largely predictable—there 
seems to be an important distinction between deterministic “automatic” 
or “automated” responses and those based on collected and analyzed 
information. 


3. “...is capable of...” 


The DoD’s definition states that a weapon system’s autonomy 
depends on whether it “can select and engage targets without further 
intervention by a human operator,” and thus the definition “includes 
human-supervised autonomous weapon systems” as a subset of 
autonomous weapon systems.*! Notwithstanding this clarity, 
subsequent writers relying on this definition often imply that human- 
supervised systems are distinct from autonomous weapon systems. To 
address this frequent misreading, this clarified definition expands the 
Directive’s “can” to “is capable of.” 


58 See Scharre & Horowitz, supra note 15, at 6 (“[T]here are no clear boundaries between 
these degrees of complexity, from ‘automatic’ to ‘automated’ to ‘autonomous’ to ‘intelligent, and 
different people may disagree on what to call any given system.”). 

59 See Sharkey, supra note 38, at 141. 

60 It bears noting that, while the predictability of a weapon system’s response to a situation 
may be a crucial consideration in a commander’s decision to deploy the weapon, see infra Part 
II.B, the nature of the environment and the predictability of a weapon system’s response to it is 
irrelevant for determining whether or not the system is autonomous. 

61 DOD DIR. 3000.09, supra note 19, at 13 (emphasis added). “[H]uman-supervised 
autonomous weapon systems” are defined as autonomous weapon systems “designed to provide 
human operators with the ability to intervene and terminate engagements, including in the event 
of a weapon system failure, before unacceptable levels of damage occur.” Id. at 14. 
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4, “... independently selecting and engaging targets.” 


As many have intuited, the fundamental distinction between semi- 
autonomous and autonomous weapon systems is that the latter are 
capable of selecting and engaging targets—which may include human 
beings, objects, or even code—without human intervention, while the 
former cannot. But what does target selection and engagement entail? 
Complete freedom to decide who lives or dies? Bounded selection or 
implementation capabilities based on preprogrammed constraints? 
Prioritizing one specific preselected target over another? 

Under this definition, the distinction turns on whether a weapon 
system uses its autonomous capabilities to select and engage a target. 
Thus, an autonomous weapon system could either operate with no 
constraints or with preprogrammed boundaries—the important fact 
would be that it gathered and processed information in the course of 
target selection and engagement. A semi-autonomous weapon system, 
in contrast, might act autonomously in functions related to target 
selection or engagement—including acquiring, tracking, prioritizing, or 
determining when or how to engage specific targets—but a human 
operator would need to take some affirmative action before it would be 
capable of both selecting and engaging a target. 


a. Specific Versus General Targets 


“Specific” is a key word here—as noted above, to the extent targets 
are preselected by human operators, the selections would need to be 
quite precise to preserve a meaningful distinction between semi- 
autonomous and autonomous weapon systems. Thus, semi- 
autonomous weapon systems might use autonomous capabilities to 
engage a particular preselected target, while autonomous weapon 
systems might be charged with engaging targets with certain 
characteristics. 

To grasp this distinction, it helps to consider the Roomba, a 
popular autonomous vacuum cleaning robot with various cleaning 
modes. In the “SPOT mode,” a human being selects a particular area for 
focused cleaning and the Roomba “moves in a slow spiral pattern over 
the soiled area.” If it encounters a wall or other object, it “will 
intelligently keep cleaning in the focused area.”® In the “CLEAN mode,” 
the Roomba “calculate[s] the room size and maximize[s] coverage-per- 
room based on information it receives through its sensors.” In each of 


62 Service & Support - FAQs: Roomba’s Cleaning Modes, I ROBOT, 
http://uksupport.irobot.com/app/answers/detail/a_id/863/~/roombas-cleaning-modes (last 
visited Aug. 17, 2014). 

63 Id. 


64 Id. 
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these modes, the Roomba is employing autonomous capabilities to 
accomplish a human-set task. And its goal in each mode is similar: to 
clean the floor. However, the human operator is selecting a “target” in 
the semi-autonomous SPOT mode, while the human operator is 
essentially charging the Roomba with selecting “targets” in a designated 
“battlefield” in the autonomous CLEAN mode. 


b. Sufficient Time for an Affirmative Action 


Some affirmative action from a human operator with regard to the 
selected target is also necessary to distinguish semi-autonomous from 
autonomous weapon systems. Where a human being merely has 
supervisory power, the weapon system is for all intents and purposes 
autonomously selecting and engaging the target. 

But what constitutes an “affirmative action”? Choosing not to 
exercise veto power can hardly be sufficient—but requiring too much 
human involvement risks setting the bar for autonomy too low and 
grouping weapon systems with widely and importantly distinct levels of 
required human involvement together. 

To evaluate the implications of this standard, it helps to consider 
weapon systems currently in use. The U.S. Phalanx Close In Weapons 
Systems (CIWS or Sea Whiz)—affectionately called “R2-D2s” by 
Americans or “Daleks” by Brits because of their barrel-like shape—are 
mounted on ships and provide a last-ditch defense against incoming 
high-speed, anti-ship missiles and low-level aircraft.% It collects data in 
real time; identifies potential targets; evaluates whether they pose a 
threat based on whether they are approaching the ship, capable of 
maneuvering to hit the ship, and traveling within a certain velocity; and 
engages them—but allows for a manual override.s The CIWS can be 
employed under the Aegis combat system,” which has four modes, 
ranging from “semiautomatic,” where a human operator controls 
decisions regarding the use of lethal force, to “casualty,” which assumes 


65 John Pike, MK 15 Phalanx Close-In Weapons System (CIWS), FED’N AM. SCIENTISTS, 
http://www.fas.org/man/dod-101/sys/ship/weaps/mk-15.htm (last updated Jan. 9, 2003); Robert 
H. Stoner, R2D2 with Attitude: The Story of the Phalanx Close-In Weapons System (CIWS), 
NAVWEAPS, http://www.navweaps.com/index_tech/tech-103.htm (last updated Oct. 30, 2009). 

66 Pike, supra note 65; Stoner, supra note 65. The U.S. Navy is currently testing a more 
advanced CIWS: the SeaRam CIWS Anti-Ship Missile Defense System “automatically detects, 
evaluates, tracks, engages, and performs kill assessment against [anti-ship missiles] and high 
speed aircraft threats in an extended self defense battle space envelope around the ship.” United 
States Navy Fact File: SeaRam Close-In Weapon System (CIWS) Anti-Ship Missile Defense System, 
US. Navy, _ http://www.navy.mil/navydata/fact_display.asp?cid=2100&tid=456&ct=2 (last 
updated Nov. 15, 2013). 

67 See Stoner, supra note 65. 
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that the human operators are incapacitated and therefore permits the 
system to use defensive force independently. 

A land-based variant of the U.S. Navy’s CIWS is the Centurion, 
which was part of the Counter-Rocket, Artillery, Mortar (C-RAM) 
initiative and was originally deployed in Iraq in 2005.6 According to 
reports, C-RAM systems have successfully intercepted over 100 
rockets.70 Similarly, Israel’s Iron Dome, which currently is the only dual 
C-RAM and Very Short Range Air Defense system, intercepted over 150 
rockets fired into Israel from the Gaza Strip prior to the July 2014 
conflict.71 In that conflict, the Iron Dome was credited with shooting 
down ninety-percent of Palestinian rockets it engaged—leading to 
“Tron Dome tourism,’ where the public, reassured by the system’s 
performance, stay outdoors to watch the shoot-downs rather than 
taking cover.” 72 

Are these semi-autonomous or autonomous weapon systems? If 
specific target selection and engagement depends on any affirmative 
action of a human operator, the U.S. C-RAM system and the Israeli Iron 
Dome would be classified as semi-autonomous. They both identify a 
threat and send a recommended response to a human operator, who 
must then decide within seconds whether to give the command to fire.”3 
Once provided with authorization, the system then determines when to 
fire and how to guide the intercepting missile to neutralize the threat. 
When a defensive system only permits a human supervisor to veto a 
determination to select and engage a target, however, it would be an 
autonomous weapon system. Thus, although the U.S. Navy's 
CIWS/Aegis system allows for a manual override,” to the extent it has 
the capability to select and engage targets with no further human 


68 Gary E. Marchant et al., International Governance of Autonomous Military Robots, 12 
COLUM. SCI. & TECH. L. REV. 272, 287 (2011). 

69 Stoner, supra note 65. Because these are used on land, where there is a greater risk to 
civilians, the C-RAM’s rounds explode either on impact or upon tracer burnout. Id. 

70 Land-Based Phalanx Weapon System Completes Mission in Iraq, NAVSEA (Feb. 16, 2012), 
http://www.navsea.navy.mil/Lists/NewsWires/DispForm.aspx?ID=12. Additionally, having a C- 
RAM system may have had a deterrent effect. As the system manager noted, “‘[w]e have logged 
numerous successful intercepts, but we don’t know how many attacks didn’t take place once they 
realized there was a defense system deployed.” Id. 

71 Iron Dome, RAFAEL ADVANCED DEF. SYS. LTD., http://www.rafael.co.il/Marketing/186- 
1530-en/Marketing.aspx (last visited Apr. 27, 2015). 

72 Dan Williams, Israel Says Iron Dome Scores 90 Percent Rocket Interception Rate, REUTERS, 
July 10, 2014, available at http://www.reuters.com/article/2014/07/10/us-palestinians-israel- 
irondome-idUSKBNOFFOXA20140710. 

73 See Iron Dome Battle Management Demonstrated, DEF. UPDATE, http://defense- 
update.com/photos/iron_dome_bms.html (last visited Apr. 27, 2015) (noting that the system 
“requests the operator’s permission to launch the missiles”); Paul Scharre, Reflections on the 
Chatham House Autonomy Conference, LAWFARE (Mar. 3, 2014, 5:34 PM), 
http://www.lawfareblog.com/2014/03/guest-post-reflections-on-the-chatham-house-autonomy- 
conference (stating that C-RAM systems do not have an autonomous mode). 

74 Pike, supra note 65. 
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intervention, it is an autonomous weapon system. But is this distinction 
between the semi-autonomous C-RAM and Iron Dome and the 
autonomous CIWS/Aegis system sensible? 

On one hand, preserving this technical distinction may have some 
positive policy side effects. It has the benefit of clarity: any given weapon 
system can be definitively classed as semi-autonomous or autonomous. 
Militaries anxious to avoid potential restrictions on autonomous 
weapon systems may favor designs requiring some affirmative human 
action over designs that merely permit a veto, resulting in a clearer 
accountability chain for every engagement. Maintaining this distinction 
may also be useful in the ban debate, as both advocates and skeptics of a 
ban on autonomous weapon systems seem willing to exclude defensive 
systems with a human “in the loop” from any potential ban. 

On the other hand, requiring an affirmative action from a human 
operator in these charged circumstances begins to look more like a 
“rubber stamp” than a considered decision to select a specific target for 
engagement.”5 To the extent the human supervisor’s affirmative action 
is rendered essentially irrelevant, given the superhuman nature of the 
required response time to an incoming threat, it seems unfair to hold 
him accountable for a decision to engage a selected target that was 
effectively made by the weapon system. 

This issue is compounded by psychological factors. First, human 
beings can place too much faith in computers, to the extent that an 
operator may trust a machine’s conclusion more than her own analysis. 
Second, while autonomous weapon systems may allow human operators 
to monitor multiple systems, studies have shown that a monitor’s 
attention declines as the number of monitored systems increases.76 
Stephen Knouse theorizes that when an individual knows her actions 
will be futile, she will lose the motivation to fulfill the duties of her 
position.” Presumably, then, the operator of a weapon systems 
subconsciously concludes that her actions are irrelevant and therefore 
might “automatically” approve the engagement of potential targets. 78 

Although there are regulatory reasons for preserving a technical 
distinction between semi-autonomous and autonomous weapon 


75 See P. W. Singer, War of the Machines: A Dramatic Growth in the Military Use of Robots 
Brings Evolution in Their Conception, 303 Sci. AM. 56, 63 (2010) (noting that, in autonomous 
defensive systems, “the operator really only exercises veto power, and a decision to override a 
robot’s decision must be made in only half a second, with few willing to challenge what they view 
as the better judgment of the machine”); see also Jeffrey S. Thurnher, No One at the Controls: 
Legal Implications of Fully Autonomous Targeting, 67 JOINT FORCE Q. 77, 83 (2012) (arguing that 
human oversight should be more than “merely a rubber stamp”). 

76 Allyson Hauptman, Autonomous Weapons and the Law of Armed Conflict, 218 MIL. L. REV. 
170, 186 (2013). 

77 Id. at 186 n.61. 

78 See id. at 186. 
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systems based on whether a human being makes any affirmative action, 
a legal definition should aim to avoid creating distinctions between 
different weapon systems that operate similarly in practice. Given this, 
and given that the primary concern with autonomous weapon systems is 
about when the decision to use lethal force is delegated to a machine, for 
a weapon system to not be effectively autonomous, the human operator 
or supervisor must have sufficient time to evaluate the nature of the 
target, its military significance, and the likely incidental effects of 
engagement.”? What constitutes sufficient time for these calculations 
will vary based on the situation. 


c. “Controllable” Versus “Fully” Autonomous Weapon Systems 


Understandably, many feel that there is an important distinction 
between human-supervised and entirely uncontrolled autonomous 
weapon systems. This intuition leads many to distinguish between 
weapon systems where human beings are “on the loop”—which many 
acknowledge are in use today—and “fully” autonomous weapon systems 
where human beings are entirely “off the loop’—which most conclude 
do not yet exist.80 

There are important policy implications to this distinction: it is 
necessary for constructing different best practices or administrative 
regulations,*! and having a human being responsible for supervising the 
actions of an autonomous weapon system might be critical for assigning 
responsibility should the system’s actions result in a violation of 
international or domestic law. Just because a weapon system may be 
capable of selecting and engaging targets without human intervention 


79 This standard for distinguishing between semi-autonomous and effectively autonomous 
weapon systems draws from the International Committee on Robot Arms Control’s list of 
minimum necessary conditions for meaningful human control. See Frank Sauer, ICRAC 
Statement on Technical Issues to the 2014 UN CCW Expert Meeting, ICRAC (May 14, 2014), 
http://icrac.net/2014/05/icrac-statement-on-technical-issues-to-the-un-ccw-expert-meeting. 
However, the Committee would require that all of their conditions be met in every situation 
where a target is engaged, which would forbid the usage of many weapon systems currently being 
used. 

80 See supra note 2; see also Michael N. Schmitt, Autonomous Weapon Systems and 
International Humanitarian Law: A Reply to the Critics, HARV. NAT'L SECURITY J. FEATURES 4-5, 
11 (2013), available at http://harvardnsj.org/wp-content/uploads/2013/02/Schmitt-Autonomous- 
Weapon-Systems-and-IHL-Final.pdf (acknowledging that “U.S. forces have operated two human- 
supervised autonomous systems for many years,” but nonetheless concluding that “fully 
autonomous weapon systems” are not being fielded by the United States); Scharre & Horowitz, 
supra note 15, at 16 (distinguishing between “autonomous weapon system[s]” and “human- 
supervised autonomous weapon system|[s]”); Lawand, supra note 8 (“A truly autonomous weapon 
system would be capable of searching for, identifying and applying lethal force to a target, 
including a human target (enemy combatants), without any human intervention or control. This 
definition connotes a mobile system with some form of artificial intelligence, capable of operating 
in a dynamic environment with no human control.”). 

81 See DOD DR. 3000.09, supra note 19, at 3 (distinguishing when and how autonomous and 
semi-autonomous weapon systems may be used). 
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does not mean it will or should be employed in that mode; indeed, there 
are significant ethical and practical arguments counseling against 
developing weapons that do not require an affirmative action from a 
human before employing lethal force.82 

But while distinguishing between human-supervised and 
unsupervised weapon systems may be important in developing best 
practices or regulations, it is irrelevant to the question of whether a 
given weapon system is autonomous.*3 Such operation does not render 
the weapon system itself any less autonomous, it just means that its 
autonomous capabilities are not being fully utilized. If a weapon system 
has the capacity to independently select and engage targets, whether 
there is a human supervisor or whether it is operated in a semi- 
autonomous mode is a question of usage—and thus regulation—and 
not of autonomy. Returning to the Roomba analogy: even if a human 
operator were to only use a Roomba in SPOT mode or only permitted it 
to use the CLEAN mode under active supervision, the Roomba would 
nonetheless remain an autonomous robot, as it would retain the 
capability to operate in the CLEAN mode. Accordingly, this definition 
does not differentiate between “controllable” and “fully” autonomous 
weapon systems. 


d. Lethal Versus Non-Lethal Weapon Systems 


“Target engagement” usually entails committing to a violent or 
disruptive action to destroy or undermine the functioning of a target. It 
includes the use of lethal force, which consists of any action—bombing, 
stabbing, infecting, and so on—that is intended to or has a substantial 
risk of causing death, serious bodily harm, or injury. “Lethal force” may 
include force directed at human beings or force directed at objects, such 


82 Writers with practical experience tend to argue that militaries are uninterested in 
developing or deploying autonomous weapon systems that do not have a human operator “on the 
loop,” as doing so “decreases the chances of weapons striking the wrong target, resulting in 
fratricide or civilian casualties, or that they simply miss their target entirely, wasting scarce and 
expensive munitions.” Paul Scharre, Autonomy, “Killer Robots,” and Human Control in the Use of 
Force—Part II, JUST SECURITY (July 9, 2014, 2:30 PM), http://justsecurity.org/12712/autonomy- 
killer-robots-human-control-force-part-ii [hereinafter Scharre, Autonomy II]; see also Werner 
J.A. Dahm, Commentary, Killer Drones are Science Fiction, WALL ST. J., Feb. 15, 2012, at All 
(noting that there is currently no military disadvantage in keeping humans involved in decisions 
regarding engagement and therefore no demand to delegate that step); Charli Carpenter, US 
Public Opinion on Autonomous Weapons, DUCK MINERVA (June 2013), 
http://www.whiteoliphaunt.com/duckofminerva/wp-content/uploads/2013/06/UMass-Survey_ 
Public-Opinion-on-Autonomous-Weapons.pdf (finding, in a 2013 survey of 1000 Americans, 
that military personnel, veterans, and individuals with family in the military were more strongly 
opposed to autonomous weapons than the general public, with the highest opposition coming 
from active duty troops). 

83 Similarly, it is inaccurate to consider an autonomous weapon system that is only operated 
in semi-autonomous modes, like the CIWS, merely a semi-autonomous weapon system. 
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as tanks, planes, ships, or buildings, whose destruction has a substantial 
risk of causing such injury to human beings. 

But what of autonomous weapon systems programmed to only 
target and engage objects whose destruction is unlikely to cause harm to 
human beings? To the extent that the concern with autonomous 
weapon systems is that human operators are inappropriately delegating 
the decision to use lethal force to machines, wouldn’t such weapon 
systems be acceptable? Put another way: Ban proponents aren’t 
concerned with autonomous weapon systems, but rather with lethal 
autonomous weapon systems. 

This definition does not, however, distinguish between lethal and 
non-lethal autonomous weapon systems. When a weapon system is 
capable of wielding destructive force, what it is preprogrammed to 
target is a question of how it is used, not of its autonomy. As a result, for 
the purposes of this definition, it is irrelevant whether a weapon system 
is engaging human or non-human targets or whether the system is being 
used for offensive or defensive purposes. *4 


D. Autonomous Weapon Systems in Use Today 


There is a nearly universal consensus, among both ban advocates 
and skeptics, that autonomous weapon systems do not yet exist.85 One 
of the more influential critiques of the pro-ban argument is that such 
weapon systems are “inevitable.” Under the clarified definition, 
however, it quickly becomes clear that autonomous weapon systems are 
not just inevitable—they already exist and have been deployed. 


84 Granted, there are different levels of risks associated with weapon systems that engage only 
nonhuman targets than those which engage human targets. Such practical distinctions may be 
quite important in discussing the regulation (as opposed to a complete ban) of autonomous 
weapon systems. See infra Part IV.C.1.b. 

85 See supra note 2. There are a few writers on the subject who acknowledge that some 
autonomous weapon systems are in use today. See MARSH, supra note 50, at 3; Ronald Arkin, 
Lethal Autonomous Systems and the Plight of the Non-Combatant, AISB Q., July 2013, at 6; 
Gubrud, supra note 18; Scharre, Autonomy II, supra note 82; see also Anderson, Reisner & 
Waxman, supra note 57, at 388-89 (“[S]everal modern highly-automated—and some would call 
them autonomous—weapon systems already exist.”); Schmitt & Thurnher, supra note 2, at 235 
(noting that the United States has operated “two ‘human-supervised’ autonomous systems for 
many years—the Aegis at sea, and the Patriot on land”). 

However, even these writers tend to limit autonomous weapons in use today to a few 
exceptions to the norm of nonautonomy. See MARSH, supra note 50, at 3 (discussing the U.K. 
Brimstone and other fire-and-forget missiles); Gubrud, supra note 18 (discussing fire-and-forget 
or lock-on-after-launch missiles); Scharre, Autonomy II, supra note 82 (discussing the Israeli 
Harpy and the PMK encapsulated torpedo mine). 

Under the clarified definition, however, it becomes clear that many different types of 
autonomous weapon systems are currently in use, many of which have not been recognized in the 
literature as such. 

86 Anderson & Waxman, supra note 4, at 2, 27. 
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1. Levels of Autonomy 


While there will necessarily be a spectrum of weapons autonomy, 
ranging from a stone to the Terminator, there are four relevant 
classifications. As pictured below, from least to most autonomous, there 
are inert weapon systems, automated weapon systems, semi- 
autonomous weapon systems, and autonomous weapon systems. 


Fig. 1: Levels of Autonomy 
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If asked to list weapons, most would likely name some variant on 
gun,” “knife,” or “sword.”87 When distinguishing between levels of 
weapon autonomy, all of these—and everything from a stone to the 
most advanced handheld firearm—could be classified as an “inert 
weapon,” as they are all objects requiring contemporaneous operation 
by a human being to be lethal. 

“Automated” weapon systems are purely reactive; although they 
may be deployed long before they engage a target, they merely follow 
commands or preprogrammed rules, without employing gathered 
information or algorithmic calculations to draw independent 
conclusions about how to react. A tripwire sentry gun will fire 
automatically after being triggered; an autonomous sentry will process 
data before firing. Thus, while both automated and autonomous 


cc 


87 One friend responded to this question: “Any object within range at a moment’s notice.” 
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weapon systems may react to certain situations in a predictable manner, 
automated weapons have no “choice” in the matter. 

Semi-autonomous weapon systems have some autonomous 
capabilities, which may include functions relevant to target selection 
and engagement, but they cannot independently both select and engage 
targets. Some such systems might identify a potential target and await 
an operator’s approval before exercising lethal force; others might 
employ autonomy in determining how to carry out a strike against a 
preselected target. However, a human operator will still need to take 
some affirmative action to select a specific target for engagement. 

Finally, autonomous weapon systems are capable of selecting and 
engaging targets based on conclusions derived from gathered 
information and preprogrammed constraints, without any 
contemporaneous decisional support by a human being. They might 
operate in structured or unstructured environments; they may be 
mobile or stationary; they may have rudimentary artificial intelligence, 
be capable of in-field learning, or even have human-level reasoning; 
they may be supervised or entirely uncontrollable. While these factors 
are germane to whether they can be used in compliance with the law of 
armed conflict and in the construction of regulatory policies, they are 
irrelevant to the question of whether a particular weapon system is 
autonomous. 


2. Incentives for Development 


A host of political, practical, and even ethical incentives appears to 
favor the development of increasingly autonomous weapon systems. Of 
course, there are also important considerations counseling against a 
headlong embrace of such weaponry.*8 The aim of this subsection is not 
to join the debate as to whether governments should employ 
autonomous weapon systems, but rather to highlight the numerous 
reasons states are investing in related research. 

First, there are powerful political incentives for a state to replace or 
augment its human forces with robotic ones. The main one is 


88 See, e.g., DOD DIR. 3000.09, supra note 19; HUMAN RIGHTS WATCH & INTL HUMAN 
RIGHTS CLINIC, HARVARD LAW SCH., ADVANCING THE DEBATE ON KILLER ROBOTS: 12 KEY 
ARGUMENTS FOR A PREEMPTIVE BAN ON FULLY AUTONOMOUS WEAPONS (2014) [hereinafter 
ADVANCING THE DEBATE], available at http://www.hrw.org/sites/default/files/related_material/ 
Advancing%20the%20Debate_8May2014_Final.pdf; LOSING HUMANITY, supra note 2; Arkin, 
supra note 85, at 4-5; Asaro, supra note 4, at 692; see also Sarah Knuckey, Scientists from 37 
Countries Call for Ban on Autonomous Lethal Targeting, JUST SECURITY (Oct. 16, 2013, 11:56 
AM), http://justsecurity.org/2013/10/16/scientists-ban-autonomous-weapons-systems (reviewing 
legal, political, strategic, moral, and ethical arguments for and against autonomous weapon 
systems). 
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summarized nicely by a Navy chief petty officer on the loss of his unit’s 
PackBot: “‘when a robot dies, you don’t have to write a letter to its 
mother.” In fact, a common argument for banning autonomous 
weapon systems is that they will make wars too easy.” If politicians 
don’t need to provide reasons for entering a conflict that justify the loss 
of their constituents’ lives, this implicit check on armed conflicts might 
disappear—along with democratic peace theory.%! Increasingly 
autonomous weapon systems are also politically appealing because they 
make life safer for soldiers: they reduce the number of soldiers exposed 
to physically and psychologically dangerous situations while 
simultaneously creating jobs for (and therefore a need to train) highly 
skilled professionals. 

Second, autonomous weapon systems may simply be more 
effective and efficient weapons. Remotely-operated weapon systems 
usually require at least a one-to-one match of operators,%2 but as semi- 
autonomous and autonomous weapon systems proliferate, human 
operators will be able to monitor larger numbers of systems—possibly 
rendering individual operators more productive, requiring less total 
staff, and increasing total force projection.” They may also extend the 
reach of any individual solider, allowing him to “see[] farther or strik[e] 
further.”°4 Additionally, to the extent that remotely-operated systems 
are vulnerable to jamming or even takeover,” increasing the weapons’ 


89 P. W. Singer, Robots at War: The New Battlefield, 33 WILSON Q. 30, 31 (2009). 

90 Id. at 47-48. Conversely, as “boots on the ground” are removed from the equation, 
politicians might find it easier to justify humanitarian interventions. See Anderson & Waxman, 
supra note 4, at 17-18 (discussing whether there is ever an “optimal” level of force). For a 
discussion of how autonomous weapon systems might make war politically easier and thereby 
affect the balance of the U.S. war power, see Rebecca Crootof, War, Responsibility, and Killer 
Robots, 40 N.C. J. INTL L. & COM. REG. (forthcoming 2015), available at http://papers.ssrn.com/ 
sol3/papers.cfm?abstract_id=2569298. 

91 See Peter M. Asaro, How Just Could a Robot War Be?, in CURRENT ISSUES IN COMPUTING 
AND PHILOSOPHY 50, 56-59 (Adam Briggle, Katinka Waelbers & Philip Brey eds., 2008); Singer, 
supra note 89, at 48. 

92 See Thurnher, supra note 75, at 79 (“It takes scores of people, from pilots to technicians to 
intelligence analysts, to operate a single tethered UAV.”). 

93 See Marchant et al., supra note 68, at 275; Singer, supra note 89, at 41 (noting that when 
human operators control two UAVs at a time, their performance levels decrease by an average of 
fifty percent—which incentivizes increased autonomy for weapon systems); see also DOD 
ROADMAP, supra note 35, at 25 (noting that personnel costs are “the greatest single cost in DoD” 
and that “strides in autonomy... have reduced the number of personnel required, but much 
more work needs to occur”). 

94 Marchant et al., supra note 68, at 275. 

95 In June 2012, a team from the University of Texas at Austin led by Professor Todd 
Humphreys used “spoofing”’—a hacker-created signal that infiltrates a drone’s GPS system—to 
take control of a drone in flight. John Roberts, EXCLUSIVE: Drones Vulnerable to Terrorist 
Hijacking, Researchers Say, FOXNEWS.COM (June 25, 2012), http://www.foxnews.com/tech/2012/ 
06/25/drones-vulnerable-to-terrorist-hijacking-researchers-say. This may be how Iran allegedly 
captured a U.S. surveillance drone in December 2011. Adam Rawnsley, Iran’s Alleged Drone 
Hack: Tough, but Possible, WIRED (Dec. 16, 2011, 6:01 PM), http://www.wired.com/2011/12/iran- 
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autonomy and eliminating the need for a continuous link between the 
weapon system and the operator might improve its efficacy. The 
possibility of autonomous “persistent stare”—longer-term surveillance 
or independent evaluation of factors on the ground—might also 
improve precision targeting and reduce collateral damage. % 

Similarly, autonomous weapon systems may be preferable to 
human soldiers in certain situations or for certain activities. Werner 
Dahm, then-Chief Scientist of the U.S. Air Force, stated in 2010 that “by 
2030 machine capabilities will have increased to the point that humans 
will have become the weakest component in a wide array of systems and 
processes.”9 To a certain extent, this is already occurring: a Defense 
Advanced Research Projects Agency official has stated that human 
beings are becoming “the weakest link in defense systems.”% 
Additionally, whereas training a soldier in a task might take days, weeks, 
or even years, weapon systems can share knowledge almost 
instantaneously.” They can quickly collect and integrate information 
from varied sources. They don’t get hungry, tired, bored, or sick. They 
are immune to biological and chemical weapons. They tackle the dirty, 
dangerous, and dull work without complaint. They can reach 
inaccessible areas and survive in inhospitable environments. They 
follow instructions to the letter. They need not be motivated by self- 
preservation, and so may “be used in a self-sacrificing manner if needed 
and appropriate.”!°° They are immune from psychological “scenario 
fulfillment,” which causes human beings to process new information in 
line with their preexisting beliefs and may sometimes result in 
“distortion or neglect of contradictory information in stressful 
situations.”!0!1 They don’t act out of fear or anger, for vengeance or 
vainglory.!°2 Ronald Arkin, a roboticist and roboethicist working to 


drone-hack-gps (reporting on Iranian claims that “Iran managed to jam the drone’s 
communication links to American operators by forcing it to shift into autopilot mode. With its 
communications down, the drone allegedly kicked into autopilot mode, relying on GPS to fly 
back to base in Afghanistan. With the GPS autopilot on, the engineer claims Iran spoofed the 
drone’s GPS system with false coordinates, fooling it into thinking it was close to home and 
landing into Iran’s clutches.”). 

96 See Arkin, supra note 85, at 1; Reeves & Johnson, supra note 4, at 26. 

97 WERNER J.A. DAHM, U.S. AIR FORCE, REPORT ON TECHNOLOGY HORIZONS: A VISION FOR 
AIR FORCE SCIENCE & TECHNOLOGY DURING 2010-2030, at 106 (2010) (Werner J.A. Dahm is the 
USS. Air Force Chief Scientist (AF/ST)). 

98 See Singer, supra note 89, at 37 (internal quotation marks omitted). 

99 Id. at 37-38. 

100 RONALD C. ARKIN, GOVERNING LETHAL BEHAVIOR IN AUTONOMOUS ROBOTS 29 (2009). 

101 Id. at 30-31. 

102 However, it is not accurate to claim that autonomous weapon systems would never rape, 
massacre, or commit other war crimes; it all depends on how they are programmed and used. See 
Charli Carpenter, “Robot Soldiers Would Never Rape”: Un-packing the Myth of the Humanitarian 
War-Bot, DUCK MINERVA (May 14, 2014), http://www.whiteoliphaunt.com/duckofminerva/2014/ 
05/robot-soldiers-would-never-rape-un-packing-the-myth-of-the-humanitarian-war-bot.html. 
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develop algorithms for an “ethical governor” for robots, argues that 
autonomous weapon systems could eventually comply with the law of 
armed conflict better than human soldiers.!03 He believes his 
decisionmaking architecture “could potentially lead to ethically superior 
robotic warriors within as few as 10 to 20 years.” 104 

Based on these and other factors, investments in unmanned 
technology are expected to grow.!°5 The DoD has published its annual 
“Unmanned Systems Integrated Roadmap,” outlining its vision for the 
“continued development, production, test[ing], training, operation, and 
sustainment of unmanned systems technology across DoD” for the next 
twenty-five years.!06 This includes the goal of “[t]ak[ing] the ‘man’ out 
of unmanned [systems]”!°7 and moving from “autonomous mission 
execution to autonomous mission performance.’ Believing that 
“[aJutonomy in unmanned systems will be critical to future conflicts 
that will be fought and won with technology,” the DoD has labeled 
increasing autonomy in unmanned systems a “high priority.” 109 

Proponents of a ban regularly highlight states’ interest in 
developing increasingly autonomous weapon systems as a reason for 
why it is necessary to prohibit them now, and skeptics of a ban often 
suggest that states’ interest renders this a particularly productive time to 
discuss the legal constraints on the use of autonomous weapon systems. 
But in their exhortations, both sides generally ignore the fact that there 
are already a number of autonomous weapon systems in use today. 


3. Autonomous Weapon Systems Today 


This subsection describes various weapon systems that would be 
classified as autonomous under the clarified definition. This list is not 
intended to be all-inclusive, but rather to provide pertinent examples of 


103 ARKIN, supra note 100, at 30. 

104 Don Troop, Robots at War: Scholars Debate the Ethical Issues, CHRON. HIGHER EDUC. 
(Sept. 10, 2012), http://chronicle.com/article/Moral-Robots-the-Future-of/134240. But see 
Sharkey, supra note 4, at 792-96 (critiquing Arkin and others for employing language that 
encourages the anthropomorphism of robotic processes by inappropriately attributing “ethical” 
capabilities to them). 

105 See Jack Browne, UAV Markets Robust Despite Declining Spending, DEF. ELECTRONICS (Feb. 
15, 2012), http://defenseelectronicsmag.com/electronic-countermeasures/uav-markets-robust- 
despite-declining-spending. 

106 DOD ROADMAP, supra note 35, at v; see also Thurnher, supra note 75, at 79 (“The 
expectation is that robots on the battlefield will form the bulk of detachments, such as infantry 
units that would be comprised of 150 human soldiers working alongside 2,000 robots.”). 

107 DOD ROADMAP, supra note 35, at 25. 

108 Id. at 66. 

109 Id. at 67. 
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a variety of types of weaponry with the ability to independently select 
and engage targets. And this is only a smattering of the publicly 
available research and development—weapon systems in development 
or even currently in use might employ greater autonomous capabilities 
than has been revealed. 


a. Autonomous Weapon Systems Operated in Semi-Autonomous or 
Human-Supervised Modes 


Many existing autonomous weapon systems are currently only 
operated in a semi-autonomous mode—much like a Roomba that is 
only used in SPOT mode. However, just as that Roomba remains an 
autonomous cleaning robot, these weapon systems are most accurately 
classified as autonomous weapon systems. 

To monitor the demilitarized zone, South Korea has allegedly 
installed SGR-Al1s: stationary, armed robots which identify potential 
human targets,!!0 voice commands to surrender, “observe” signs of 
surrender, and react after consultation with human supervisors in 
nearby command centers.!!! Although it is not clear whether or not they 
have been deployed, the Super aEgis II has also been designed for use in 
monitoring the demilitarized zone: it consists of a gun tower that can 
“find and lock on to a human-sized target in pitch darkness at a distance 
of up to 1.36 miles,” uses anything from a 12.7 mm caliber machine gun 
to a surface-to-air missile to fire, and can be mounted on the ground or 
on a moving vehicle.1!2 Both the SGR-A1 and Super aEgis II may be set 
to modes where they can select and engage targets with no human 
involvement or oversight. 113 

Israel’s Guardium Unmanned Ground Vehicle is a boxy, car-like 
vehicle that patrols the Israel/Gaza border. It can be operated remotely 
or programmed to act autonomously, “both driving itself and 


110 The SGR-A1 identifies every human being who enters the demilitarized zone as an enemy, 
on the grounds that the individual has entered a prohibited zone. Samsung Techwin SGR-A1 
Sentry Guard Robot, GLOBALSECURITY.ORG, http://www.globalsecurity.org/military/world/rok/ 
sgr-al.htm (last visited Apr. 27, 2015). These robots are reportedly assisted by technology based 
on Microsoft’s Kinect—originally developed for use with the Xbox 360. See Brian Ashcraft, 
Microsoft’s Kinect Is Now Guarding the Korean Border, KOTAKU (Feb. 3, 2014, 8:00 AM), 
http://kotaku.com/microsofts-kinect-is-now-guarding-the-korean-border-1514792443?utm_ 
campaign=Socialflow_Kotaku_Facebook&utm_source=Kotaku_Facebook&utm_medium=Social 
flow. 

111 See Jon Rabiroff, Machine Gun-Toting Robots Deployed on DMZ, STARS & STRIPES (July 12, 
2010), http://www.stripes.com/news/pacific/korea/machine-gun-toting-robots-deployed-on- 
dmz-1.110809; Samsung Techwin SGR-A1 Sentry Guard Robot, supra note 110. 

112 Loz Blain, South Korea’s Autonomous Robot Gun Turrets: Deadly from Kilometers Away, 
GIZMAG (Dec. 7, 2010), http://www.gizmag.com/korea-dodamm-super-aegis-autonomos-robot- 
gun-turret/17198. 

113 Id.; Samsung Techwin SGR-A1 Sentry Guard Robot, supra note 110. 
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responding to obstacles and events,”!!4 and can conduct surveillance 
and use non-lethal or lethal force.!!5 While the Guardium appears to be 
employed in a semi-autonomous mode, it may also be able to select and 
engage targets autonomously. 116 

Lastly, as noted above, the U.S. Navy’s Aegis control system, used 
to identify and eliminate incoming ballistic threats, has four modes. 
When in “casualty” mode, the human operators are assumed to be 
incapacitated and the system is able to use defensive force 
independently.1!7 Various generations of the CIWS/Aegis systems are 
now used in the navies of twenty-three U.S.-allied nations.!!8 Nor are 
USS. allies the only ones with this technology: Russia’s AK-630, the 
Netherlands’ Goalkeeper, and Italys DARDO are similar weapon 
systems.1!9 Germany also employs the NBS MANTIS, a land-based, 
“fully automated air defence system” that will “detect, track and shoot 
the projectiles within a close range of the target base.” 120 


b. Offensive Autonomous Weapon Systems 


Autonomous weapon systems have also been designed for offensive 
situations—and many currently existing weapon systems might be 
employed more offensively if states so desire. As Dahm has noted, “[I]t’s 
not technology that has held us back from fully autonomous military 
strikes—from a purely technical perspective, it has been possible for 
some time to conduct them.” 121 

While many fire-and-forget or lock-on-after-launch missiles are 
guided by a human deployer, sometimes by radio or lasers, and are 
therefore merely semi-autonomous, numerous other ones 
independently select and engage targets. The U.K. Brimstone is touted 
as “a fully autonomous, fire-and-forget, anti-armour weapon, effective 


114 Adam May, Phantom on the Fence, ISR. DEF. FORCES BLOG (Oct. 9, 2012, 5:24 PM), 
http://www. idf.il/1283-17082-EN/Dover.aspx. Such responsive action is “likely alerting a 
command center to the presence of something suspicious, not opening fire without notifying a 
human operator first.” John Reed, Israel’s Killer Robot Cars, FOREIGN POL’Y (Nov. 20, 2012, 5:33 
PM), http://foreignpolicy.com/2012/11/20/israels-killer-robot-cars. 

115 Enguard! Introducing the Guardium UGV, DEF. UPDATE, http://defense-update.com/ 
products/g/guardium.htm (last visited Apr. 27, 2015). 

116 Id. 

117 Marchant et al., supra note 68, at 287. 

118 Stoner, supra note 65. 

119 See also Scharre & Horowitz, supra note 15, at 12 (“At least 30 nations employ or have in 
development at least one system of this type....”); id. app. B (describing autonomous weapon 
systems in use today). 

120 NBS MANTIS Air Defence Protection System, Germany, ARMY-TECHNOLOGY.COM, 
http://www.army-technology.com/projects/mantis (last visited Apr. 27, 2015). 

121 Dahm, supra note 82. 
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against all known and projected armoured threats.”!22 When in search 
mode, “Brimstone’s mmW seeker searches for targets in its path, 
comparing them to a known target signature in its memory. The missile 
automatically rejects returns which do not match...and continues 
searching and comparing until it identifies a valid target.” 123 It is unclear 
whether the operator programs the Brimstone to seek out a specific 
radar target or ones with certain characteristics; to the extent the latter is 
true, the Brimstone would be an autonomous weapon system. 

The Israeli Harpy Loitering Weapon “detects, attacks and destroys 
enemy radar emitters.”!24 Unlike a semi-autonomous weapon, “[t]he 
person launching the Harpy does not know[] which particular radars 
are to be engaged, only that radars that meet the Harpy’s programmed 
parameters will be engaged.”!25 Nor is Israel the only country with the 
Harpy: it has been sold to Turkey, South Korea, India, and China. 126 

The United States is funding research in related technology: in late 
2012 the U.S. Air Force Research Laboratory awarded Boeing Phantom 
Works a $10 million contract for the Dominator, a long-endurance 
UAV which will be able to autonomously conduct intelligence, 
surveillance, and reconnaissance missions and would potentially have 
strike capabilities.127 It is intended to carry Textron Common Smart 
Submunitions, warheads which can independently search for and 
engage targets in a two-acre area after deployment. 128 

There are also sea-based autonomous weapon systems. Unlike 
other types of mines, which usually detonate automatically, 
encapsulated torpedo mines “are a type of sea mine that, when activated 
by a passing ship, instead of exploding, open a capsule which then 
releases a torpedo that engages a target.”!2° The eventual target of the 
torpedo is entirely unknown by the human deployer; instead, “the mine 
is selecting and engaging targets on its own.”130 Russia and China are 
both employing the PMK-2 encapsulated torpedo mine today.13! 


122 ROYAL AIR FORCE, AIRCRAFT & WEAPONS 87 (Brian Handy ed., 2007) (U.K.), available at 
http://www.raf.mod.uk/rafcms/mediafiles/0186cc2a_1143_ec82_2ef2bffff37857da.pdf. 

123 Id. There are numerous other missiles with similar autonomous target-selection 
capabilities. See MARSH, supra note 50, at 3. 

124 Harpy Loitering Weapon, ISR. AEROSPACE INDUSTRIES, http://www.iai.co.il/2013/16143- 
16153-en/IAL.aspx (last visited Apr. 27, 2015). 

125 Scharre, Autonomy I, supra note 41. 

126 Id. 

127 Bill Carey, Boeing Phantom Works Develops ‘Dominator’ UAV, AIN ONLINE (Nov. 2, 2012, 
10:30 AM), _ http://www.ainonline.com/aviation-news/ain-defense-perspective/2012-11-02/ 
boeing-phantom-works-develops-dominator-uav. 

128 Id. 

129 Scharre, Autonomy I, supra note 41. 

130 Id. 

131 Id. 
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Last but not least, there are autonomous ground units. In addition 
to the aforementioned Israeli Guardium, Russia has recently unveiled 
Platform-M, a “universal combat platform.”!32 It is a multipurpose 
weapon system, designed “for gathering intelligence, for discovering and 
eliminating stationary and mobile targets, for firepower support, for 
patrolling and for guarding important sites.”133 Its “weapons can be 
guided, it can carry out supportive tasks and it can destroy targets in 
automatic or semiautomatic control systems”!34+—which presumably 
means, under the clarified definition, it can operate in autonomous or 
semi-autonomous modes. 


+ * 


This Part proposed a clarified definition for autonomy in weapon 
systems, which focuses on a weapon systems capability for 
independently selecting and engaging targets based on conclusions it 
draws from gathered information and preprogrammed constraints. 
With this new definition it quickly becomes clear that, contrary to the 
general consensus, autonomous weapon systems are not weapons of the 
future: they exist and are in use today. This fact has profound 
implications for the debate on banning such weaponry. The crucial 
question is not whether we should ban some imagined Terminator or 
Hal, but rather how we can best use law to regulate weapons with a 
proven track record. 


II. ARE AUTONOMOUS WEAPON SYSTEMS INHERENTLY UNLAWFUL? 


Proponents of a ban make a number of important moral, political, 
and strategic arguments, but their primary legal claim is that 
autonomous weapon systems would never be able to comply with the 
law of armed conflict. More specifically, they argue that autonomous 
weapon systems will be unable to distinguish between lawful and 
unlawful targets; will be unable to conduct in bello proportionality 
assessments; and may not accord with the Martens Clause, which, they 
argue, requires new technology to comply “the principles of humanity” 
and “the dictates of the public conscience.” Some question whether 
states will be able to hold individuals accountable for war crimes 
committed by autonomous weapon systems; still others posit that, as a 


132 Andrew Tarantola, Russia’s Military Is Getting Killer Wall-E Robot Soldiers in 2018, 
GIZMODO (July 15, 2014, 11:40 AM), http://gizmodo.com/russias-military-is-getting-killer-wall- 
e-robot-soldier- 1604674629. 

133 Id. 

134 Id. 
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matter of law, the decision to kill another human being can never be 
delegated to a machine. 

Skeptics of a ban tend to respond to these arguments on their own 
terms, highlighting controversial or inaccurate interpretations or 
arguing that the law of armed conflict is sufficiently flexible to address 
new issues.!35 Skeptics also often dispute advocates’ pessimistic 
conclusions about autonomous weapon systems’ future capabilities. 136 

Both sides of the debate take a fundamental, inaccurate 
assumption—that autonomous weapon systems are not in use today—as 
true. But given that such weaponry has already been integrated into 
states’ militaries with little critique, many ban advocates’ legal 
arguments lose their force. Autonomous weapon systems are currently 
being lawfully employed, demonstrating that this class of weaponry is 
not inherently unlawful. 


A. Compliance with the Distinction Requirement 


One of the most fundamental rules in the law of armed conflict is 
that of distinction.!37 Parties to a conflict must distinguish between 
lawful targets—combatants, military objectives, and civilians directly 
participating in hostilities—and unlawful targets—civilians, civilian 
objects, and persons hors de combat.'38 Accordingly, parties are 
prohibited from using inherently indiscriminate weapons, which are 
usually defined either as weapons that cannot be directed at lawful 
targets or as weapons whose effects cannot be controlled. Additionally, 
any given attack in an armed conflict cannot be indiscriminate: it must 
be directed at a lawful target and cannot utilize indiscriminate weapons 
or methods of warfare. 

At present, most agree that autonomous weapon systems are 
incapable of distinguishing between combatants and civilians. Doing so 
requires a complicated assessment of various factors, and there are 
many gray zones that bewilder even well-trained human soldiers. For 
example, civilians taking direct part in hostilities are lawful targets, but 
armed civilians acting as law enforcement are not. Robotic systems may 


135 See, e.g., Schmitt & Thurnher, supra note 2, at 233. 

136 See, e.g., id. at 234. 

137 This customary rule is codified in multiple treaties, but most notably in Article 48 of the 
First Additional Protocol. See Protocol Additional to the Geneva Conventions of 12 August 1949, 
and Relating to the Protection of Victims of International Armed Conflicts (Protocol J), art. 48, 
adopted June 8, 1977, 1125 U.N.T.S. 3 [hereinafter First Additional Protocol]. 

138 An individual is hors de combat if he “is in the power of an adverse Party”; “clearly 
expresses an intention to surrender”; or “has been rendered unconscious or is otherwise 
incapacitated by wounds or sickness, and therefore is incapable of defending himself” so long as 
the individual “abstains from any hostile act and does not attempt to escape.” Id. art. 41(2). 
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be better than humans at certain tasks, but they are notoriously inept at 
recognizing objects, let alone distinguishing between lawful and 
unlawful targets.139 Presumably, autonomous weapon systems will be 
deemed to be compliant with the distinction requirement whenever 
their determinations regarding who or what is a lawful target are 
comparable to non-autonomous systems. But whether autonomous 
weapon systems will ever be able to make such distinctions is currently a 
matter of hypothetical debate: Arkin thinks robots may be able to 
comply with the distinction requirement in as few as ten years;!40 
Sharkey is skeptical that they ever will be able to do so, let alone do so in 
the near future. !41 

But this does not mean, as many pro-ban advocates conclude, that 
autonomous weapon systems are per se unlawful.1#2 First, certain 
autonomous weapon systems might be used in compliance with the 
distinction requirement based on what they are capable of targeting. 
The Israeli Harpy, for example, only “sees” and thus only targets radars. 
Second, as Michael Schmitt points out, a weapon’s inability to 
distinguish between lawful and unlawful targets simply limits where and 
when such weaponry may be lawfully deployed. He observes: 


Not every battlespace contains civilians or civilian objects. When 
they do not, a system devoid of any capacity to distinguish protected 
persons and objects from lawful military targets can be used without 
endangering the former.... The inability of the weapon systems to 
distinguish bears on the legality of their use in particular 
circumstances ..., but not their lawfulness per se.143 


Therefore, “[a]n autonomous weapon system only violates the 
prohibition against weapons incapable of being directed at a lawful 
target if there are no circumstances, given its intended use, in which it 
can be used discriminately.”!44 Schmitt notes the Iraqi use of SCUD 
missiles against troops in the civilian-free Iraqi desert during the 1990- 
1991 Gulf War as an example of indiscriminating weapons that were 
nonetheless used discriminately (and therefore lawfully).145 

The SCUD example helps clarify Schmitt’s point, but his 
argument—that autonomous weapons can be used in a discriminating 


139 See Noel E. Sharkey, Towards a Principle for the Human Supervisory Control of Robot 
Weapons, 2014 POLITICA & SOCIETA 305 (2014) (comparing relative strengths and weaknesses of 
human beings and autonomous systems). 

140 Troop, supra note 104. 

141 See Sharkey, supra note 4, at 788-89; see also ADVANCING THE DEBATE, supra note 88, at 5; 
The Scientists’ Call... to Ban Autonomous Lethal Robots, ICRAC, http://icrac.net/call (last visited 
Apr. 27, 2015). 

142 See, e.g., LOSING HUMANITY, supra note 2, at 30. 

143 Schmitt, supra note 80, at 11. 

144 Id. at 13. 

145 Id. at 10. 
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fashion, notwithstanding their inability to discriminate between 
combatants and civilians—could have been even stronger had he 
discussed certain weaponry now being so used. The Korean SGR-A1, for 
example, is an autonomous weapon system being lawfully used in a 
semi-autonomous mode to patrol the demilitarized zone. Given the 
unique characteristics of the zone, where all human beings in certain 
areas can be presumed to be combatants or civilians directly 
participating in hostilities, the SGR-A1 might even be lawfully employed 
in its autonomous mode. 146 

In response, ban advocates fall back to policy reasons for banning 
autonomous weapon systems, arguing that “[n]arrowly constructed 
hypothetical cases in which fully autonomous weapons could lawfully be 
used should not be employed to legitimize the weapons or stand in the 
way of a ban because the cases do not alter the underlying concerns 
about the use of such weapons.”!47 As a practical matter, it may be 
difficult to restrict the use of autonomous weapon systems to areas with 
few civilians, like the Korean demilitarized zone.'48 But this is a policy 
response to a legal argument, and such cases are far from “hypothetical.” 

A better legal argument for the per se unlawfulness of autonomous 
weapon systems would focus not on whether they could distinguish 
between combatants and civilians, but instead on whether they could 
distinguish between active and wounded combatants. The former are 
lawful targets, the latter are not, and both are likely to be found in areas 
of active hostilities. 

While stronger, this argument is only relevant to anti-personnel 
autonomous weapon systems—not autonomous weapon systems 
generally. The CIWS, for example, targets only missiles and incoming 
planes (which can fairly be presumed to be piloted by hale combatants), 
and thus sidesteps this issue entirely.149 In short, given that some 


146 See Jeffrey S. Thurnher, The Law That Applies to Autonomous Weapon Systems, 17 ASIL 
INSIGHTS (Jan. 18, 2013), — http://www.asil.org/insights/volume/17/issue/4/law-applies- 
autonomous-weapon-systems (“There may be situations in which an autonomous weapon system 
could satisfy this rule with a considerably low level ability to distinguish between civilian and 
military targets. Examples would include during high intensity conflicts against declared hostile 
forces or in battles that occur in remote regions, such as underwater, deserts, or areas like the 
Demilitarized Zone in Korea.”). 

147 ADVANCING THE DEBATE, supra note 88, at 8. 

148 See id. at 9. 

149 In a recent paper, Eliav Lieblich and Eyal Benvenisti imaginatively question whether 
autonomous weapon systems will impermissibly limit the opportunity for combatants to 
surrender and violate the prohibition on ordering that no quarter be given. Eliav Lieblich & Eyal 
Benvenisti, The Obligation to Exercise Discretion in Warfare: Why Autonomous Weapon Systems 
Are Unlawful 35-39 (Oct. 2014) (working paper), available at http://papers.ssrn.com/sol3/ 
papers.cfm?abstract_id=2479808. As they acknowledge, there is currently no requirement under 
the law of armed conflict to grant enemy combatants an opportunity to surrender. Id. at 35-36. 
However, even if such a requirement existed, as with the discretion argument, this concern is only 
relevant to anti-personnel autonomous weapon systems. 
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autonomous weapon systems are capable of being used discriminately 
and with controllable effects, they are not as a class per se unlawfully 
indiscriminate. 


B. Compliance with the in Bello Proportionality Requirement 


A second foundational customary requirement of the law of armed 
conflict is that of in bello proportionality.!5° As articulated in the First 
Additional Protocol to the 1949 Geneva Conventions, the 
proportionality requirement prohibits as indiscriminate “[a]n attack 
which may be expected to cause incidental loss of civilian life, injury to 
civilians, damage to civilian objects, or a combination thereof, which 
would be excessive in relation to the concrete and direct military 
advantage anticipated.” 151 

The commander who authorizes an attack is responsible for 
making the proportionality analysis.152 Given the difficulty human 
beings have in making the proportionality analysis, given its inherent 
subjective and contextual elements,153 military commanders enjoy a 
great deal of discretion in subsequent evaluations of their decisions. 
When assessing whether a military action has met the proportionality 
standard, evaluators focus on the knowledge available to the 
commander at the time of the strike, and the “Rendulic Rule” suggests 
that a “reasonable commander” standard be applied when determining 
liability. 154 

Ban advocates argue that autonomous weapon systems will likely 
never be able to qualitatively analyze, let alone weigh, the expected 
military advantage of a particular attack and the associated potential 


150 Jus ad bellum is the law governing the commencement of hostilities; jus in bello is the law 
governing the conduct of hostilities. There are separate ad bellum and in bello proportionality 
requirements, but disputes regarding the ability of autonomous weapon systems to be used in 
compliance with the proportionality requirement generally focus on the in bello rule, which 
requires a proportionality analysis for each individual attack in an armed conflict. 

151 First Additional Protocol, supra note 137, art. 51(5)(b). 

152 Schmitt, supra note 80, at 20. 

153 See John Fabian Witt, Two Conceptions of Suffering in War, in KNOWING THE SUFFERING 
OF OTHERS: LEGAL PERSPECTIVES ON PAIN AND ITS MEANINGS 129, 147-50 (Austin Sarat ed., 
2014) (noting the difficulty of assigning military objectives and human suffering objective values, 
particularly given the relevance of the context within which the engagement occurs). 

154 See Prosecutor v. Galić, Case No. IT-98-29-T, Judgment and Opinion, § 58 (Intl Crim. 
Trib. for the Former Yugoslavia Dec. 5, 2003) (“In determining whether an attack was 
proportionate it is necessary to examine whether a reasonably well-informed person in the 
circumstances of the actual perpetrator, making reasonable use of the information available to 
him or her, could have expected excessive civilian casualties to result from the attack.”); see also 
Eric Talbot Jensen, Essay, Unexpected Consequences from Knock-on Effects: A Different Standard 
for Computer Network Operations?, 18 AM. U. INT'L L. REV. 1145, 1181-83 (2003) (discussing the 
history of the Rendulic Rule). 
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harm to civilians !55—particularly insofar as the proportionality analysis 
requires “distinctively human judgement.”!56 Nor, they argue, can 
responsibility for making the proportionality analysis rest with the 
human deployer, as the proportionality analysis might change between 
deployment and execution. 157 

In response, ban skeptics suggest that parameters for making in 
bello proportionality analyses could “in theory” be preprogrammed into 
autonomous weapon systems, allowing them to make basic, 
conservative proportionality assessments and contact a human operator 
in more complex situations.!5§ The U.S. military already employs a 
“collateral damage estimate methodology,” a procedure “for 
determining the likelihood of collateral damage to objects or persons 
near a target,” the result of which in turn determines who on the chain 
of command must authorize an attack causing collateral damage.!59 
Presumably, an autonomous weapon system might be preprogrammed 
to be able to select and engage targets when there is little to no chance of 
collateral damage, but be required to seek out human approval for 
attacks with a higher likelihood of collateral damage. 160 

The fact that autonomous weapon systems are already being used 
with little critique, however, suggests that they can be operated in 
compliance with the proportionality requirement.'*! But, just like the 
proportionality analysis itself, when an autonomous weapon system can 
be so employed is fact-dependent and context-specific. 

One lawful possibility involves the use of autonomous weapon 
systems in semi-autonomous modes. A commander might determine 
that a specific attack complies with the proportionality requirement and 
authorizes an autonomous weapon system, operating in a semi- 
autonomous mode, to engage a given target. This is how some 
autonomous weapon systems, like the Israeli Guardium and Korean 
SGR-A1, are reportedly used today. In these situations, the autonomous 
weapon system is being used lawfully, notwithstanding its inability to 
make its own proportionality determination. 


155 ADVANCING THE DEBATE, supra note 88, at 6. 

156 Heyns Report, supra note 5, at 14. 

157 ADVANCING THE DEBATE, supra note 88, at 6-7. 

158 Schmitt, supra note 80, at 20-21. 

159 Id. at 19. 

160 See John S. Canning, You’ve Just Been Disarmed. Have a Nice Day!, 28 IEEE TECH. & SOC’Y 
MAG. 13, 13-15 (2009) (recommending that autonomous weapon systems primarily target 
weapons, rather than their wielders, and proposing a “dial-a-level” of autonomy when it is 
necessary to target human beings). 

161 See Benjamin Kastan, Autonomous Weapons Systems: A Coming Legal “Singularity”?, 2013 
J.L. TECH. & POL’y 45, 62 (2013) (concluding that, in certain situations, autonomous weapon 
systems may be able to be lawfully used notwithstanding their inability to conduct proportionality 
analyses). 
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Another lawful possibility would occur when a commander 
determines that a variety of potential engagements in a given battlespace 
and limited temporal span would comply with the proportionality 
requirement and authorizes an autonomous weapon system to select 
and engage targets within those preprogrammed constraints. Certain 
fire-and-forget or lock-on-after-launch weapon systems like the Israeli 
Harpy or U.K. Brimstone are employed in this manner: although they 
are autonomously selecting and engaging targets, the limited temporal 
span between their deployment and potential engagements permits a 
commander to take responsibility for the proportionality analysis— 
provided, of course, that the commander takes what may occur during 
the time between deployment and engagement into account.162 To do 
so, of course, the commander must be trained in the weapon system’s 
capabilities, limitations, and destructive potential. 

The most difficult question is whether autonomous weapon 
systems, operating on extended missions or in unforeseen 
environments, could be adequately preprogrammed to comply with the 
proportionality requirement. At present, there are no autonomous 
weapon systems able to engage in such long-range missions, but states 
are investing in the development of weapon systems with such 
capabilities. The Energetically Autonomous Tactical Robot (EATR) 
project, for example, is developing an autonomous robotic platform that 
could forage for plant biomass to refuel itself over long-range, long- 
endurance missions, allowing for an attenuated relationship between the 
deployment of a ground system and its actions (and, incidentally, 
leading to one of the more entertaining press releases of all time). 163 

Where there is a significant temporal span between deployment 
and potential engagements or where the weapon system is venturing 
into unknown territory, a commander could not reasonably take 
responsibility for conducting the proportionality requirement; too many 
of the conditions the weapon system might face would be 
unpredictable.164 At some point in the future, it may be possible to 
preprogram autonomous weapon systems with sufficient constraints to 
satisfy the proportionality analysis requirement—and perhaps even 


162 Cf. O'Connell, supra note 4, at 234 (arguing that a new norm of international law, requiring 
a close temporal distance between force deployment and target engagement, is necessary to “keep 
a human conscience” in the decision). 

163 See Press Release, Cyclone Power Techs., Cyclone Power Technologies Responds to 
Rumors About “Flesh Eating” Military Robot (July 16, 2009), available at 
http://www.robotictechnologyinc.com/images/upload/file/Cyclone%20Power%20Press%20 
Release%20EATR%20Rumors%20Final%2016%20July%2009.pdf (insisting that the robot would 
be “strictly vegetarian” in response to “the public’s concern about futuristic robots feeding on the 
human population”). 

164 See ADVANCING THE DEBATE, supra note 88, at 6-7. 
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permit more precise analyses than human beings currently make.165 
However, given their current inability to discriminate between lawful 
and unlawful targets and their inability to weigh the military advantage 
of a particular engagement against anticipatable harm to unlawful 
targets, at present autonomous weapon systems cannot now be lawfully 
deployed on extended missions or in unknown environments. 

Autonomous weapon systems are already employed in various 
ways and in compliance with the in bello proportionality requirement. 
The fact that they cannot be so used in other circumstances is not a legal 
argument for a complete ban on such weaponry; rather, it is in 
argument in favor of regulation illuminating when and how 
autonomous weapon systems can be lawfully used. 


C. The Inapplicability of the Martens Clause 


The Martens Clause appears in the first article of the First 
Additional Protocol to the 1949 Geneva Conventions. It provides: “In 
cases not covered by this Protocol or by other international agreements, 
civilians and combatants remain under the protection and authority of 
the principles of international law derived from established custom, 
from the principles of humanity and from the dictates of the public 
conscience.” 166 

The Martens Clause has been described by the International Court 
of Justice as customary international law, binding on all states, and as 
“an effective means of addressing the rapid evolution of military 
technology.” 167 But how it is to be understood and applied is still hotly 
debated. Interpreted narrowly, the Martens Clause implies that, in the 
absence of positive treaty law, customary international law, the 
principles of humanity, and public conscience should guide states’ 
actions.!16 Given a broader reading, the Martens Clause elevates 
compliance with “the principles of humanity” and “the dictates of the 
public conscience” to additional, independent legal requirements. 169 

Some ban advocates suggest that, given the lack of international 
law explicitly regulating autonomous weapon systems, the Martens 


165 Schmitt, supra note 80, at 20-21. The U.S. Naval Research Laboratory, for example, is 
currently testing a Cognitive Robot Abstract Machine, which is supposed to be able to process 
new information and make decisions faster and better than human beings. It can classify and 
identify objects, run algorithms to test various action-based scenarios, and ultimately adopt the 
“best” scenario. See Hauptman, supra note 76, at 185. 

166 First Additional Protocol, supra note 137, art. 1(2). 

167 Legality of the Threat or Use of Nuclear Weapons, Advisory Opinion, 1996 I.C.J. 226, 257 
(July 8) [hereinafter Nuclear Weapons]. 

168 See Tyler D. Evans, Note, At War with the Robots: Autonomous Weapon Systems and the 
Martens Clause, 41 HOFSTRA L. REV. 697, 713-19 (2013). 

169 See id. at 713-14, 716 (discussing the current debate over the scope of the Martens Clause). 
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Clause is relevant.!70 Favoring the broad reading, they conclude that 
autonomous weapon systems are only lawful under the Clause if they do 
not violate “the principles of humanity” and “the dictates of the public 
conscience.”!71 They then question whether autonomous weapon 
systems meet these standards, noting that robotic systems do not 
experience compassion—which the International Committee of the Red 
Cross (ICRC) has stated is vital to the principle of humanity—and citing 
Charli Carpenter’s 2013 study finding that, of those with an opinion, 
sixty-eight percent of surveyed Americans oppose the use of fully 
autonomous weapon systems. 172 

Skeptics of a ban, employing a narrowed reading, respond that the 
Martens Clause is not “an overarching principle that must be considered 
in every case,” but rather is “a failsafe mechanism meant to address 
lacunae in the law.”173 Even assuming for the sake of argument that the 
broad reading is the best reading, some question whether there is 
sufficient public consensus to ban autonomous weapon systems under 
the Martens Clause. 174 

Arguments for a ban based on the Martens Clause are not very 
strong, standing alone, not least because the Clause is usually employed 
to supplement other legal reasoning.!75 Additionally, the commonly 
cited empirical evidence of public concern regarding autonomous 
weapon systems (Carpenter’s 2013 survey) was based on a definition of 
autonomy for weapon systems that could be understood as implying 
that they operated without any restraint. Participants were asked how 
they felt about “the trend toward using completely autonomous [robotic 
weapons/lethal robots] in war,” which were described as “robotic 
weapons that can independently make targeting and firing decisions 
without a human in the loop.”!76 Unsurprisingly, most opposed using 
weapons which had no clear restrictive mechanisms;!”” it remains 
unknown what percentage of participants would oppose the use of 
weapons that “can independently make targeting and firing decisions” 
but are subject to preprogrammed constraints. 

Furthermore, the existence and use of autonomous weapon 
systems further weakens the persuasive power of legal arguments 


170 ADVANCING THE DEBATE, supra note 88, at 15. 

171 Id. 

172 Id. 

173 Schmitt & Thurnher, supra note 2, at 275. 

174 Evans, supra note 168, at 727-28 (noting that even the authors of Losing Humanity concede 
that there is currently no consensus regarding whether autonomous weapon systems would 
violate these principles). 

175 See id. at 717-18. 

176 Carpenter, supra note 82, at 1 (the wording was varied to measure the extent the term used 
affected public sentiment). 
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grounded in the Martens Clause. First, there is some question as to 
whether the Clause is even applicable, as autonomous weapon systems 
are governed by both the law of armed conflict and many other treaty 
and customary international legal regimes.178 Second, autonomous 
weapon systems in use today have not drawn criticism on the grounds 
that they violate the “principles of humanity” and the “dictates of the 
public conscience.” 


D. Extraneous Accountability Issues 


Who is to be held accountable when an autonomous weapon 
system violates the law of armed conflict? Its operator? Its operator's 
commander? Its programmer—or, more complicating, its team of 
programmers? Its manufacturer? The autonomous weapon system 
itself? Many have concluded that none of these possibilities are 
satisfactory or fair,!”° and some suggest that this provides an additional 
reason why autonomous weapon systems would be unlawful. 18° 

But while the issue of accountability raises fascinating moral, 
philosophical, policy, and even domestic and international legal 
questions, it does not bolster arguments that autonomous weapon 
systems are inherently unlawful. Whether a weapon is per se unlawful is 
not, and has never been, based on whether an individual can be held 
accountable for violations following from its use.18! 


178 See Rebecca Crootof, The Varied Law of Autonomous Weapon Systems, in NATO ALLIED 
COMMAND TRANSFORMATION, AUTONOMOUS SYSTEMS: ISSUES FOR DEFENCE POLICY MAKERS 
(Andrew P. Williams & Paul D. Scharre eds.) (forthcoming 2015), available at 
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2569322 (describing other legal regimes— 
including international human rights law, the law of the sea, the law of outer space, and the law 
regarding state responsibility for private actors—that provide guidance on the lawful usage of 
autonomous weapon systems). 

179 See, e.g., Robert Sparrow, Killer Robots, 24 J. APPLIED PHIL. 62 (2007). Of course, legal 
regimes regularly make liability determinations that may seem “unfair” from a philosophical 
standpoint, usually for various policy reasons—as evidenced by any domestic strict liability 
regime. Compare id. (concluding that, because no one can ethically be assigned liability for the 
actions of an autonomous system, they cannot be used), with Kastan, supra note 161, at 69 
(“[G]eneral philosophical objections to applying accountability either to the humans directing 
[autonomous weapon systems] or to the systems themselves stand in opposition to long-standing 
principles of legal accountability.”). 

180 See, e.g., O'Connell, supra note 4, at 236 (concluding that the use of autonomous weapon 
systems “would conflict with the historical, legal, and moral understanding that killing should be 
based on a good-faith understanding of real necessity and carried out by someone who may be 
held accountable for a wrong decision”); Lawand, supra note 8 (“If responsibility cannot be 
determined as required by [international humanitarian law], is it legal or ethical to deploy such 
systems?”). 

181 Rather, weapons are only per se unlawful if they cannot be used discriminately, if they 
cause superfluous injury, or if they are specifically banned (usually by treaty). See infra Part 
HI.B.1-2. 
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Furthermore, autonomous weapon systems in use today have 
arguably contributed to breaches of the law of armed conflict; that has 
not prevented states from continuing to use them. The Aegis system, for 
example, may have played a part in the Flight 655 tragedy.182 But it is 
still employed by Australia, Japan, South Korea, Norway, Spain, and the 
United States, and many other countries use other, similar systems. 
Significantly, neither states, scholars, nor the most vocal critics of 
autonomous weapon systems have argued that these ongoing uses are 
per se unlawful. 

Constructing a liability regime may require a more carefully 
calibrated definition for autonomous weapon systems than the one 
suggested by this Article. To best assign liability, it will be important to 
consider the finer distinctions between an Aegis system, operated in its 
semi-autonomous mode, and a Terminator-like system that enjoys 
near-full autonomy. Determining how best to assign individual liability 
for different types of violations will be a regulatory headache—and a 
project for a different paper.!83 For this Article, it is sufficient to note 
that while accountability questions undoubtedly pose a challenge, it is 
neither an insurmountable one nor a strong legal argument for a ban. 


E. The Nonexistent Nondelegation Principle 


Asaro suggests that there is a principle in the law of armed conflict 
that “the authority to decide to initiate the use of lethal force cannot be 
legitimately delegated to an automated process, but must remain the 
responsibility of a human with the duty to make a considered and 
informed decision before taking human lives.”184 Asaro explicitly casts 
his argument as a legal one, suggesting that this norm is on par with the 
requirement that all attacks are discriminating and proportional. 185 

However, Asaro’s claim is actually a moral one—or, at best, an 
argument for the creation of a new legal norm. As a matter of 
international law, there is no treaty provision requiring that a human 


182 See Singer, supra note 89, at 40. 

183 Many have begun outlining the issues involved in constructing liability regimes. See, e.g., 
Tim McFarland & Tim McCormack, Mind the Gap: Can Developers of Autonomous Weapons 
Systems Be Liable for War Crimes?, 90 INT'L L. STUD. 361 (2014); Thilo Marauhn, Professor, Justus 
Liebig Univ., Presentation at the CCW Expert Meeting on Lethal Autonomous Systems: An 
Analysis of the Potential Impact of Lethal Autonomous Weapons Systems on Responsibility and 
Accountability for Violations of International Law (May 13-16, 2014), available at 
http://www.unog.ch/80256EDD006B8954/%28httpAssets%29/35FEA015C2466A57C1257CE400 
4BCA51/$file/Marauhn_MX_Laws_SpeakingNotes_2014.pdf; see also Kastan, supra note 161, at 
65-81 (discussing U.S. domestic issues related to assigning liability for the actions of autonomous 
weapon systems). 

184 Asaro, supra note 4, at 689. 

185 Id. at 687-88. 
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being make “a considered and informed decision” before taking another 
human life, and to the extent there is evidence of a customary norm, it 
points the other direction. Many automated weapons now in use fail to 
accord with this principle, but they are rarely critiqued on that basis. 
Although he acknowledges this fact, Asaro never grapples with the blow 
the current, accepted use of many automated weapons deals to his 
theory—much less the current, accepted use of autonomous weapon 
systems. 186 


The legal arguments made by pro-ban advocates rest on the 
assumption that autonomous weapon systems do not yet exist—and ban 
skeptics who respond to these claims tend to take the same assumption 
for granted. But autonomous weapon systems are currently being 
lawfully used, demonstrating that this class of weaponry is not 
inherently unlawful. 

Acknowledging this fact fundamentally alters the terms and stakes 
of the debate. First, proponents of a ban might do well to acknowledge 
that their strongest arguments for a ban are moral, policy, or strategic 
ones. Second, rather than debating the inherent lawfulness of 
autonomous weapon systems, those interested in using legal means to 
limit the use and development of such weaponry—whose number 
includes both advocates and skeptics of a ban—could more productively 
spend their time discussing how best to regulate weapon systems now in 
use. 


HI. CAN AUTONOMOUS WEAPON SYSTEMS BE SUCCESSFULLY BANNED? 


Human beings are remarkably ingenious at developing new means 
and methods of killing each other. Throughout history, new weapons 
have been critiqued for being too ignoble, gruesome, or haphazard—but 
only some of these weapons have been successfully banned. 

Scholars on both sides of the ban debate have mined historical 
examples for evidence that states will or will not be able to successfully 
ban autonomous weapon systems. Proponents of a ban point to widely- 
adhered-to prohibitions on chemical weapons and permanently 
blinding lasers.187 Skeptics, noting failed attempts to ban crossbows and 


186 See id. at 694. 

187 See, e.g., ADVANCING THE DEBATE, supra note 88, at 24; Memorandum from the Mines 
Action Can. to Convention on Conventional Weapons Delegates (May 2014) [hereinafter Mines 
Action Can. Memorandum], available at https://bankillerrobotscanada.files. wordpress.com/2014/ 
05/international-piv-memo-final.pdf. 
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aerial bombardment, respond that weapon bans are generally 
ineffective.188 But there is little analysis of which of these historical 
examples is most aptly analogized to autonomous weapon systems. To 
the extent scholars do discuss similarities and distinctions between 
earlier banned weaponry and autonomous weapon systems, they tend to 
ignore the crucial fact that these weapons are already integrated into 
states’ armed forces. 

Based on commonly cited attempted and enacted weapon bans,189 
this Part teases out eight traits which seem to increase the likelihood 
that a given ban will be successful—which is to say, both enacted and 
effective at limiting the usage of the banned weapon. Based on these 
characteristics, it concludes that states are unlikely to negotiate a 
successful ban on autonomous weapon systems. 


A. Qualities of a Successful Ban 


Every weapon ban success story is the product of a unique 
combination of factors, including the weapon’s inherent traits, its recent 
usage, prevailing moral and ethical concerns, and the status and 
interests of concerned states. It is tempting—and not inaccurate—to 
conclude that each ban is sui generis, the product of hard work and 
happy coincidence. 

That being said, successful bans do seem to have certain qualities in 
common, and failed attempts to ban a class of weapon share a number 
of contrary characteristics. At the risk of oversimplifying the complex 
interactions that have resulted in successful bans, this Section attempts 
to tease out some common qualities. 19 


1. Weapons Causing Superfluous Injury or Unnecessary Suffering 


The Saint Petersburg Declaration of 1868, which banned explosive 
bullets and provided incentive for the eventual banning of expanding 
bullets, proclaimed that the object of warfare “would be exceeded by the 
employment of arms which uselessly aggravate the sufferings of disabled 
men, or render their death inevitable” and that the use of such weapons 


188 See, e.g., Reeves & Johnson, supra note 4. 

189 The Appendix reviews various attempted and enacted bans on different classes of weaponry 
and types of warfare, including crossbows, aerial bombardment, submarines, nuclear weapons, 
cluster munitions, anti-personnel landmines, biological weapons, chemical weapons, and 
permanently blinding lasers. 

190 This list is not intended to be exhaustive, nor does it discuss the likely relative weight of the 
different factors. 


2015] THE KILLER ROBOTS ARE HERE 1885 


would therefore “be contrary to the laws of humanity.”!! This 
prohibition has since evolved into a customary rule of armed conflict, 
reiterated in numerous treaties.!92 Technically, because their use is 
already forbidden, there should be no need to explicitly ban weapons 
causing superfluous injuries—they are already per se unlawful. 

In practice, states naturally disagree about which weapons qualify 
as per se unlawful on these grounds, and advocates of nearly all modern 
attempted and enacted bans have argued that the weapon in question 
causes superfluous injury or unnecessary suffering. Accordingly, while 
claims that specific weapons cause superfluous injuries have 
undergirded weapon bans since the 1868 Declaration, an argument that 
a given weapon is per se unlawful on this ground does not significantly 
increase the likelihood of a ban being concluded or of its ultimate 
effectiveness. 

That being said, if injuries caused by a weapon inspire widespread 
public concern and civil society engagement, they may indirectly 
contribute to the likelihood of a ban’s success.!93 Additionally, a ban 
might clarify whether or not there is consensus as to whether the use of 
a given weapon generally causes superfluous injury or unnecessary 
suffering. 


2. Inherently Indiscriminate Weapons 


Because indiscriminate attacks are prohibited, inherently 
indiscriminate weapons—which includes weapons that cannot be 
directed at a military objective and weapons whose effects cannot be 
controlled—are per se unlawful. This is both a practical and 
humanitarian prohibition: to the extent a weapon’s effects cannot be 
directed or controlled, it threatens both the users’ troops and nearby 
civilians. 

Ban advocates regularly argue that the weapon in question is 
inherently indiscriminate. But, like weapons causing superfluous 
injuries, states can disagree in good faith on this question. For example, 
many advocates of the Mine Ban Convention argue that anti-personnel 
landmines are per se unlawfully indiscriminate;!%4 the United States has 
responded that “smart” mines, which could destruct or deactivate 


191 Declaration Renouncing the Use, in Time of War, of Certain Explosive Projectiles, Nov. 
29-Dec. 11, 1868, 18 Martens Nouveau Recueil (ser. 1) 474, 138 Consol. T.S. 297. 

192 Rule 70. Weapons of a Nature to Cause Superfluous Injury or Unnecessary Suffering, INTL 
COMMITTEE RED CROSS, http://www.icrc.org/customary-ihl/eng/docs/v1_rul_rule70 (last visited 
Apr. 27, 2015). 

193 See infra Part III.B.7. 

194 See, e.g., Why the Ban, INT'L CAMPAIGN TO BAN LANDMINES, http://www.icbl.org/en-gb/ 
problem/why-the-ban.aspx (last visited Apr. 27, 2015). 
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automatically after a certain period of time or at the end of active 
hostilities, would not be.!95 Nor will near-universal recognition that a 
weapon is inherently indiscriminate ensure that states will be able to 
conclude a treaty ban, as evidenced by nuclear weapons. 

It is worth noting that, just by existing, a ban may contribute to its 
own effectiveness. In situations where a weapon is not universally 
recognized as indiscriminate at the time a ban is concluded, the passage 
of the ban itself may lead to the recognition of the weapon’s 
indiscriminate nature. The Mine Ban Convention is largely credited 
with the increasing stigmatization of anti-personnel landmines,!% and 
advocates of the Convention on Cluster Munitions hope that it will have 
a similar effect.197 Similarly, states’ military manuals now tend to 
describe biological weapons as prohibited because they are 
indiscriminate—not because their use is forbidden by treaty law. 198 


3. Ineffective Weapons 


Discussing the Hague Conference’s bans of various forms of 
weaponry, one scholar reflected: 


Such destructive weapons, for instance, as the high explosive shell, 
the shrapnel, mines or torpedoes, were retained as legitimate means 
of warfare, whereas the inefficient expanding and explosive bullets 
were condemned along with the perfectly useless free balloons. The 
proceedings of the Hague Conference demonstrate rather that a 
weapon will be restricted in inverse proportion, more or less, to its 
effectiveness; that the more efficient a weapon or method of warfare 
the less likelihood there is of its being restricted in action by the rules 
of war. 199 


This realist analysis has been borne out to some degree in recent 
practice: many credit the relative success of the biological and chemical 


195 See Emily Alpert, Why Hasn’t the U.S. Signed an International Ban on Landmines?, L.A. 
TIMES BLOG (Apr. 5, 2012, 4:45 AM), http://latimesblogs.latimes.com/world_now/2012/04/mine- 
treaty-us-ottawa-convention.html. 

196 See, e.g., Rachel Good, Note & Comment, Yes We Should: Why the U.S. Should Change Its 
Policy Toward the 1997 Mine Ban Treaty, 9 Nw. J. INT'L HUM. RTS. 209, 210 (2011). 

197 See, e.g, HUMAN RIGHTS WATCH, TWELVE FACTS AND FALLACIES ABOUT THE 
CONVENTION ON CLUSTER MUNITIONS 6 (2009) [hereinafter HRW TWELVE FACTS], available at 
http://www.hrw.org/news/2009/04/14/twelve-facts-and-fallacies-about-convention-cluster- 
munitions. 

198 See Practice Relating to Rule 71. Weapons That Are by Nature Indiscriminate, INTL 
COMMITTEE RED CROSS, https://www.icrc.org/customary-ihl/eng/docs/v2_cha_chapter20_rule71 
(last visited Apr. 27, 2015) (citing sources). 

199 M.W. ROYSE, AERIAL BOMBARDMENT AND THE INTERNATIONAL REGULATION OF WARFARE 
131-32 (1928); see also Bonnie Docherty, The Time Is Now: A Historical Argument for a Cluster 
Munitions Convention, 20 HARV. HUM. RTS. J. 53, 59-61 (2007). 
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weapon bans with the fact that they are weapons with little military 
utility for powerful states, especially relative to newer weaponry.200 

Equally, once a weapon has demonstrated its utility, it will be 
difficult to convince states to cease using it. Most failed bans share two 
crucial characteristics in common, one of which is that they attempted 
to prohibit the use of extremely effective weapons. 

However, the conclusion that the law will always bow to effective 
weapons?!—which is to say, that effective weapons can never be 
banned—is not entirely accurate. Chemical weapons have proven their 
utility in many situations, yet the chemical weapons ban is one of the 
more effective ones in existence today.202 The Mine Ban Convention is 
another possible exception to the general rule: despite its many 
arguments that anti-personnel mines have military value, the United 
States has now committed to ceasing to produce or acquire banned 
mines in the future with an eye toward compliance with the 
Convention.203 


4. Other Means Exist for Accomplishing the Same Military Objective 


Certain weapons may efficiently accomplish a military objective: 
permanently blinding lasers, for example, can be used to disable enemy 
troops. However, when a state may use another weapon—say, 
temporarily blinding lasers—to accomplish the same military objective 
by other means, it will be more willing to relinquish the other one.204 

Again, the converse is also true: states will be less willing to 
voluntarily surrender a weapon which provides the only means of 


200 See, e.g., Mark J. Osiel, Obeying Orders: Atrocity, Military Discipline, and the Law of War, 
86 CALIF. L. REV. 939, 992 n.199 (1998). The United States unilaterally renounced its biological 
weapons research after concluding they were of limited military effectiveness. See Docherty, supra 
note 199, at 60. 

201 See Hays Parks, Submarine-Launched Cruise Missiles and International Law: A Response, 
103 U.S. NAVAL INST. PROC. 120, 120 (1977) (“In the consideration of any new weapon, 
technology must yield to the law, or the law to technological change. Where a weapon has proved 
effective, the latter usually has occurred.”). 

202 See RICHARD M. PRICE, THE CHEMICAL WEAPONS TABOO 5 (1997); Jozef Goldblat, The 
Biological Weapons Convention—An Overview, 318 INTL REV. RED CROSS 251, 264 (1997) 
(describing chemical weapons as “predictable, capable of producing immediate effects and, 
consequently, useful in combat”). 

203 Press Release, Office of the Press Sec’y, The White House, Fact Sheet: Changes to U.S. Anti- 
Personnel Landmine Policy (June 27, 2014) [hereinafter Fact Sheet], available at 
http://www.whitehouse.gov/the-press-office/2014/06/27/fact-sheet-changes-us-anti-personnel- 
landmine-policy. But see Docherty, supra note 199, at 60 (arguing that the success of the Mine 
Ban Convention is due in part to the fact that anti-personnel landmines have declining military 
utility). 

204 See MARSH, supra note 50, at 2; cf. Hauptman, supra note 76, at 192 (“Laws that focus more 
on methodology, rather than outcomes, are more likely to gain adherence because state will not 
feel as if their legitimate options for attaining military victory have been prohibited.”). 
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accomplishing certain goals. The second crucial characteristic most 
failed weapon bans have in common is that they attempted to ban 
weapons which, at least at the time, were unique in their ability to wreak 
a certain type of devastation or accomplish certain goals. 


5. Clear and Narrowly Tailored Prohibitions 


Bans are more likely to be effective if they clearly describe what 
weapons are and are not permitted. The Chemical Weapons 
Convention, for example, has an extensive Annex on Chemicals 
delineating why and what specific agents are prohibited.205 Clear 
descriptions increase the effectiveness of a ban at three critical points in 
time. First, states want to understand precisely what capabilities they are 
foregoing at the time of treaty ratification. Second, because states know 
precisely what is forbidden, they will be less likely to use or invest 
research monies in developing related weapons. Finally, in most cases, 
other states will be able to easily identify violations of the ban.2% For 
treaty regimes with enforcement mechanisms, this will make formal 
enforcement easier; for those without, states and civil society can still 
engage in informal enforcement actions, such as naming and shaming. 

Furthermore, many bans owe their existence and success to the fact 
that they are narrowly tailored. The prohibition on blinding laser 
weapons does not prohibit the development or use of temporarily 
blinding weapons, such as dazzlers.207 Additionally, it only forbids the 
use of weapons where the primary purpose is to cause blinding—it 
explicitly does not cover blinding which occurs “as an incidental or 
collateral effect of the legitimate military employment of laser systems, 
including laser systems used against optical equipment.”208 The 
Biological Weapons Convention excepts agents that have “justification 
for prophylactic, protective or other peaceful purposes,” 20 and the Mine 


205 Convention on the Prohibition of the Development, Production, Stockpiling and Use of 
Chemical Weapons and on Their Destruction, Annex on Chemicals, Jan. 13, 1993, S. Treaty Doc. 
No. 103-21, 1974 U.N.T.S. 45 [hereinafter Chemical Weapons Convention]. 

206 There is reason to believe, however, that this generalization will not hold true for 
autonomous weapon systems, no matter how precisely they are defined. See infra Part III.B. 

207 See Protocol on Blinding Laser Weapons (Protocol IV) art. 1, Oct. 13, 1995, S. Treaty Doc. 
No. 105-1, 2024 U.N.T.S. 167. In 2006, the United States employed laser dazzlers in Iraq to 
temporarily incapacitate drivers who ignored warnings at checkpoints. James Rainey, A Safer 
Weapon, With Risks, L.A. TIMES, May 18, 2006, http://articles.latimes.com/2006/may/18/world/ 
fg-laser18. 

208 Protocol IV, supra note 207, art. 3. 

209 Convention on the Prohibition of the Development, Production and Stockpiling of 
Bacteriological (Biological) and Toxin Weapons and on Their Destruction art. I, Apr. 10, 1972, 26 
USS.T. 583, 1015 U.N.T.S. 163 [hereinafter Biological Weapons Convention]. 
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Ban Convention excludes anti-tank mines equipped with anti-handling 
devices from its definition of “anti-personnel mines.” 210 

Broader, vague bans have had far less success. The 1907 Hague 
Declaration’s prohibition of “the launching of projectiles and explosives 
from balloons” may have enjoyed greater ratification rates had it not 
also banned similar launchings “by other new methods of a similar 
nature.”211 Additionally, the United States—originally one of the key 
initiators of a ban on anti-personnel landmines—may have joined the 
Mine Ban Convention had the other state parties been willing to make 
certain exceptions that restricted its application.2!2 


6. Prior Regulation 


The existence of a prior treaty or customary regulation, either on 
the same form of weaponry or on a similar type or class of weapons, 
may encourage states to conclude and abide by a later, stronger treaty 
ban. The success of the chemical weapons ban has been traced to the 
prohibitions on the use of poison or poisoned weapons,?3 and the 
relatively effective anti-personnel landmine ban was preceded by a 
regulatory treaty. 

If, however, new technological developments have increased the 
military utility of a weapon, the existence of a prior, related treaty will 
not be decisive. States were relatively willing to ban aerial bombardment 
by balloon; they were far less willing to ban aerial bombardment after 
the invention of the airplane. 


7. Public Concern and Civil Society Engagement 


The existence and growing success of the Mine Ban Convention 
and the Convention on Cluster Munitions has been widely attributed to 
the active participation of nongovernmental organizations and other 
civil society representatives, both prior to, during, and after the treaty 


210 Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti- 
Personnel Mines and on Their Destruction art. 2, Sept. 18, 1997, 2056 U.N.T.S. 211 [hereinafter 
Mine Ban Convention] (defining an “anti-handling device” as a device “which activates when an 
attempt is made to tamper with or otherwise intentionally disturb the mine”). 

211 Declaration (XIV) Prohibiting the Discharge of Projectiles and Explosives from Balloons, 
Oct. 18, 1907, 36 Stat. 2439, 1 Bevans 739 [hereinafter 1907 Declaration]. 

212 See David E. Sanger, U.S., in Shift, Says It May Sign Treaty to Ban Land Mines, N.Y. TIMES, 
Sept. 15, 1997, http://www.nytimes.com/1997/09/15/world/us-in-shift-says-it-may-sign-treaty-to- 
ban-land-mines.html. 

213 See PRICE, supra note 202, at 15. 
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negotiations.2/4 Harold Koh argues that these “transnational norm 
entrepreneurs” can play an important role in altering states’ interests. 215 
Some have suggested that such actors will likely play an increasingly 
influential role in the future.216 

That being said, this factor is far from decisive. Despite 
longstanding crusades for a ban on nuclear weapons by a plethora of 
prominent individuals and international and domestic organizations— 
including direct action organizations, environmental groups, consumer 
protection groups, professional associations, and political and religious 
organizations—a ban on nuclear weapons still seems out of reach. 


8. Sufficient State Commitment 


All of the aforementioned characteristics will help determine 
whether a state will be willing to voluntarily renounce its right to use an 
otherwise lawful weapon. And, ultimately, states’ commitment is the 
best indicator of a weapon bans’ success: without it, no ban can be 
concluded, let alone be effective. 

But state commitment to a ban is not purely a product of, say, 
whether the weapon is uniquely effective or abhorrent to the general 
public. It may also depend on the number of other states willing to 
renounce the weapon. Thus, there may be a snowball effect: as more 
states publically declare their support for a ban, more states may be 
willing to support it. The status of supporting states is also relevant to a 
ban’s success. If a treaty ban is ratified by the vast majority of states in 
the international community, but not by states that produce or use the 
weapon in question, it would be difficult to argue that the ban is 
successful.2!7 Finally, a ban with a small number of initial state parties 
could potentially become effective over time, if its language evolves into 
customary international law and thereby becomes binding on all states. 
However, without widespread subscription or buy-in from the most 
relevant states, a treaty is unlikely to contribute significantly to the 
development of relevant customary norms. 


214 See, e.g., Richard Falk & Andrew Strauss, On the Creation of a Global Peoples Assembly: 
Legitimacy and the Power of Popular Sovereignty, 36 STAN. J. INTL L 191, 199-201 (2000). 

215 Harold Hongju Koh, Address, The 1998 Frankel Lecture: Bringing International Law Home, 
35 HOUS. L. REV. 623, 656-63 (1998) (discussing the role of these norm entrepreneurs with regard 
to the Mine Ban Treaty). 

216 See Eric Talbot Jensen, The Future of the Law of Armed Conflict: Ostriches, Butterflies, and 
Nanobots, 35 MICH. J. INT’L L. 253, 282 (2014). 

217 See Reeves & Johnson, supra note 4, at 29 n.55. 
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B. Implications for Autonomous Weapon Systems 


The prior section identified eight traits often associated with 
successful bans. Of these, only one characteristic—civil society 
engagement—suggests that a ban on autonomous weapon systems 
would be enacted or effective; the others are inconclusive or currently 
weigh against the likelihood of a successful ban. 

Acknowledging that autonomous weapon systems have already 
been integrated into states’ armed forces affects the analysis of many of 
these factors. First, there is evidence that autonomous weapon systems 
can be highly effective in certain circumstances, such as where faster- 
than-human reaction times are necessary. Second, many of the 
objectives accomplished by autonomous weapon systems could not be 
similarly achieved by other means.2!8 Because autonomous weapon 
systems now in use are uniquely effective, it will be difficult to convince 
states to voluntarily relinquish them.2!9 Finally, there has not been any 
great public outcry over the current deployment of autonomous weapon 
systems. 

Two aspects of the clarity factor undermine the likelihood of a 
successful ban on autonomous weapon systems. First, because the 
possibilities for their use is far from understood, let alone exhausted, 
states are unlikely to agree to prohibit them. Second, even were a ban 
enacted, autonomous weapon systems are unusual in that it may not be 
possible to determine if and when the ban is violated. As evidenced by 
conflicting reports on the Korean SGR-AI, it is difficult for outside 
observers to determine the autonomous capabilities of any given system. 
As a result, only those deploying a weapon system will know if it is 
semi-autonomous or autonomous, supervised or unsupervised. 


218 Because the ban on permanently blinding lasers does not prohibit the use of lasers generally 
or even the use of temporarily blinding lasers, it does not prevent states from accomplishing 
similar military objectives. Therefore, contrary to many assertions, this ban provides little 
precedential value for a ban of autonomous weapon systems. See Schmitt & Thurnher, supra note 
2, at 281 n.158; see also Hauptman, supra note 76, at 195 n.89. Contra Mines Action Can. 
Memorandum, supra note 187 (discussing the relevance of the ban on permanently blinding 
lasers); see also O'Connell, supra note 4, at 15; Q&A on Fully Autonomous Weapons, HUM. RTS. 
WATCH (Oct. 21, 2013), http://www.hrw.org/news/2013/10/21/qa-fully-autonomous-weapons. 

Additionally, while they had not yet been used in combat, the effects of blinding lasers were 
fairly predictable, see Evans, supra note 168, at 721-22, 731, and unnecessarily harmful, see 
ADVANCING THE DEBATE, supra note 88, at 17—unlike the potential effects of as-yet-undeveloped 
autonomous weapon systems. 

219 Cf. MARSH, supra note 50, at 4 (“At the very least, if the Brimstone and other ‘fire and 
forget’ missiles are assumed to fall under the definition used by the Campaign, then consider 
effort may need to be expended on explaining why they are not to be considered militarily 
essential.”); see also ADVANCING THE DEBATE, supra note 88, at 18 (noting that, once autonomous 
weapon systems technology is developed, “many countries would be reluctant to give it up, 
especially if their competitors were deploying it”). 
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While states do seem interested in how autonomous weapon 
systems might be regulated,220 there is hardly sufficient state 
commitment at present to suggest a ban would be successful. At the 
May 2014 Experts Meeting, only a few states called for an outright 
ban.221 Nor is there any evidence that the military powerhouses 
currently investing in autonomous weapon research and development 
would join such a treaty. In fact, the United Kingdom opposes any such 
ban.222 

Other factors do not weigh heavily one way or the other. There is 
nothing intrinsic to autonomous weapon systems that would cause 
superfluous injury or unnecessary suffering; a bullet fired by an 
autonomous sentry robot causes the same amount of injury as one fired 
by a human sentry.?23 Certainly some autonomous weapon systems 
could be designed to cause such harm—just as certain bullets could be 
designed to cause excessive damage—but that does not mean the class of 
weapons as a whole should be banned. And, as discussed above, even 
assuming that autonomous weapons cannot discriminate between 
lawful and unlawful targets does not make them per se unlawfully 
indiscriminate. As evidenced by autonomous weapon systems in use 
today, they can be employed in a discriminating manner and with 
controlled effects.224 Nor is there any directly relevant prior regulation. 

The only characteristic which seems to increase the likelihood of a 
successful ban is the strong civil society campaign, comprised of a 
growing coalition of legal scholars, philosophers, human rights activists, 
scientists, Nobel laureates, and diverse organizations.225 Due to their 
efforts, “killer robots” are no longer perceived as the stuff of science 


220 See Knuckey, supra note 7 (noting that, at the May 2014 Experts Meeting, all thirty states 
making opening statements welcomed the conversation, many states took an active role in 
discussions, and no state argued that it was the wrong time or forum to be addressing these 
issues—instead, some states proposed that the issue be considered additionally in other forums). 

221 See Bonnie Docherty, Taking on “Killer Robots,” JUST SECURITY (May 23, 2014, 11:30 AM), 
http://justsecurity.org/10732/guest-post-killer-robots (reporting that five states have called for a 
ban); Knuckey, supra note 7 (stating that Ecuador, Egypt, and Pakistan called for a ban on fully 
autonomous weapon systems). 

222 Owen Bowcott, UK Opposes International Ban on Developing ‘Killer Robots,’ THE 
GUARDIAN, Apr. 13, 2015, http://www.theguardian.com/politics/2015/apr/13/uk-opposes- 
international-ban-on-developing-killer-robots. 

223 See George R. Lucas, Jr., Automated Warfare, 25 STAN. L. & POL’Y REV. 317, 330 n.24 
(2014). Contra Mark Gubrud, The Principle of Humanity in Conflict, ICRAC (Nov. 19, 2012), 
http://icrac.net/2012/11/the-principle-of-humanity-in-conflict (suggesting that there “is a human 
right not to be killed on the decision of machines”); Wendell Wallach, Terminating the 
Terminator: What to Do About Autonomous Weapons, SCI. PROGRESS (Jan. 29, 2013), 
http://scienceprogress.org/2013/01/terminating-the-terminator-what-to-do-about-autonomous- 
weapons (arguing that permitting machines to make life-or-death decisions is mala in se—evil in 
itself—like the use of rape or biological weaponry). 

224 See supra Part ILA. 

225 See ADVANCING THE DEBATE, supra note 88, at 24. 
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fiction; rather, they are the subject of Special Rapporteur reports,226 
international meetings,227 and U.S. congressional briefings.22° These 
norm entrepreneurs are working to frame the debate on autonomous 
weapon systems to increase the likelihood of a successful ban.229 
Whether this will be sufficient, especially when balanced against the 
other factors, has yet to be seen. 

Perhaps the most critical determinant of an autonomous weapon 
system ban’s success will be how narrowly tailored the chosen definition 
is. If the official threshold for autonomy in a weapon system is set 
excessively high, it will exclude weapon systems with autonomous 
capabilities currently in use today or likely to be developed in the near 
future, which would increase the likelihood of state subscription. 
However, this would grant states wide latitude in using weapon systems 
capable of independently selecting and engaging targets, which 
undermines the original impetus for a ban.230 Additionally, as Nicolas 
Marsh points out, there is a paradoxical effect to a definition of 
autonomy that would require machines to engage in human-level 
reasoning: “A key motivation for the Campaign [to Stop Killer Robots] 
is that robots cannot make the ethical and contextual assessments that 
humans can. However, a robot with a decisionmaking capability as 
advanced as human cognition could presumably make such 
judgments.”23! Alternatively, if the threshold for autonomy in weapon 
systems is set too low, states will be extremely unlikely to ratify the 
treaty, as they will be unwilling to give up what they perceive as 
necessary technology. 

Finally, regardless of how autonomous weapon systems are 
defined, a ban might exclude certain types or uses no one seems to find 
objectionable, like autonomous weapon systems now employed for 
purely defensive purposes or “smart” weapons which better distinguish 
between lawful and unlawful targets.232 While such exclusions will likely 
increase ratification rates, they would also undermine many ban 
advocates’ goals, such as limiting military investment in weapon 
autonomy and preventing the proliferation of autonomous weapon 
systems. 


226 Heyns Report, supra note 5. 

227 Chairperson Simon-Michel Report, supra note 6. 

228 Tim Starks, Killer Robots, Outer Space and Defense Spending Bill in Week Ahead, ROLL 
CALL (July 14, 2014, 7:30 AM), http://blogs.rollcall.com/five-by-five/killer-robots-outer-space- 
and-defense-spending-bill-in-week-ahead/?dcz=. 

229 See, e.g., ARTICLE 36, KEY AREAS FOR DEBATE ON AUTONOMOUS WEAPONS SYSTEMS 
(2014); ADVANCING THE DEBATE, supra note 88; LOSING HUMANITY, supra note 2; MARSH, supra 
note 50; Mines Action Can. Memorandum, supra note 187. 

230 See supra Part I.B.4. 

231 MARSH, supra note 50, at 3. 

232 See Horowitz & Scharre, supra note 41; Scharre, supra note 73. 
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++ * 


Developments in weapons technology often challenge basic 
precepts of the existing law governing armed conflicts. The crossbow 
changed the common understanding of who could be a valuable solider; 
submarines, airplanes, and cyberattacks expanded the scope of potential 
battlefields; nuclear weapons raised the possibility of total war and even 
human-caused human extinction. Autonomous weapon systems 
similarly raise important questions as to the necessary level of human 
involvement in decisions to use lethal force, leading many to call for 
their complete prohibition. 

But autonomous weapon systems share few qualities with weapons 
successfully banned in the past. They are not inherently indiscriminate, 
and they can be designed such that they do not cause superfluous injury 
or unnecessary suffering. Moreover, as evidenced by autonomous 
weapon systems in use today, they can be used lawfully, effectively, and 
carry out tasks that could not be otherwise achieved. As a result, states 
are unlikely to conclude—let alone comply with—a treaty banning their 
use, unless the ban is so narrowly tailored that it effectively defines 
autonomous weapon systems out of existence. 

All in all, this analysis suggests that states and other parties 
interested in the governance of autonomous weapon systems should 
focus not on banning them, but rather on determining how best to 
regulate their use. 


IV. How SHOULD AUTONOMOUS WEAPON SYSTEMS BE REGULATED? 


“Killer robots” are here, and they are here to stay. But that does not 
imply that efforts to galvanize state interest in their governance have 
been wasted—quite the opposite. All interested parties should take 
advantage of the current momentum to focus on how best to regulate 
this new and swiftly-evolving weaponry. Accordingly, after discussing 
the need for intentional international lawmaking and its myriad 
benefits, this Part concludes with concrete suggestions regarding the 
optimal structure and content of regulations for autonomous weapon 
systems. 


A. Intentional International Regulation Is Needed—Now 


Some ban skeptics suggest that existing international law is 
currently sufficient to regulate autonomous weapon systems, and so 
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states need not take any formal action.?33 To a certain degree, this is 
true. If states do nothing, autonomous weapon systems will continue to 
be governed by the law of armed conflict, a patchwork of relevant law 
from other legal regimes, and various types of soft law.234 To the extent 
areas of ambiguity remain, international law will evolve to address 
them. 235 

But a laissez-faire approach sacrifices the current opportunity to 
channel state practice and direct the emergence of new, relevant 
international law. In the absence of an international legislative-like body 
that can weigh ethical, policy, and other considerations to create law 
proactively, international law evolves in a reactionary way, based 
retrospectively on state action. Such precedent is based primarily on the 
specific circumstances of the situation and the state interests at that 
point in time—not on long-term implications or considerations of how 
a given action could be construed in the future. Absent some form of 
intentional law-making focused on the question of how autonomous 
weapon systems should be employed, law will be relegated to describing 
how technological innovations are—and thus may be—employed. 

Nor is the law of armed conflict set in stone. It provides general 
guidance on how any weapon may be lawfully used, but aside from 
requiring states to conduct reviews to prevent the deployment of 
unlawful weapons, the law of armed conflict imposes few limitations 
on weapons research or proliferation.237 In the absence of intentional 
regulation, the unchecked development of autonomous weapon systems 
may well pose a significant threat to fundamental humanitarian 
principles and protections—and, by extension, to human lives. 
Weapons systems with increasingly autonomous capabilities may allow 
for an increasingly attenuated temporal and geographic link between a 
human being’s decision to deploy a weapon and the use of lethal force, 
which in turn may undermine current conceptions about what 
constitutes compliance with the distinction requirement, the 


233 See, e.g., Schmitt & Thurnher, supra note 2, at 233. 

234 See Anderson & Waxman, supra note 4, at 22; Crootof, supra note 178; Marchant et al., 
supra note 68, at 306-13. 

235 See, e.g., Anderson & Waxman, supra note 4, at 27. 

236 See First Additional Protocol, supra note 137, art. 36 (“In the study, development, 
acquisition or adoption of a new weapon, means or method of warfare, a High Contracting Party 
is under an obligation to determine whether its employment would, in some or all circumstances, 
be prohibited by this Protocol or by any other rule of international law applicable to the High 
Contracting Party.”). 

Many argue that this responsibility is one of customary law, as it “flows logically from the 
truism that States are prohibited from using illegal weapons, means and methods of warfare or 
from using weapons, means and methods of warfare in an illegal manner.” Intl] Comm. of the Red 
Cross, A Guide to the Legal Review of New Weapons, Means and Methods of Warfare: Measures to 
Implement Article 36 of Additional Protocol I of 1977, 88 INT’L REV. RED CROSS 931, 933 (2006). 

237 See ADVANCING THE DEBATE, supra note 88, at 4. 
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proportionality requirement, and other relevant law. 

Autonomous weapon systems are multiplying as governments 
invest huge sums in relevant research and development. However, we 
have not yet passed an event horizon such that law is doomed to follow 
technological innovations in weapons autonomy. It is still possible to 
proactively employ legal means to channel how this new technology 
develops and is used, but the window of opportunity is closing. The time 
to act is now. 


B. Benefits to Intentional International Regulation 


There are a number of benefits to the premeditated regulation of 
autonomous weapon systems, foremost among which is that it will 
channel research and state practice by limiting what lawful options are 
available. Additionally, the very act of setting forth definitions and rules 
regarding development, usage, or transfer improves the likelihood that 
autonomous weapon systems will be used in accordance with the law, 
because states will have a better understanding of what conduct is 
permissible. Clarified rules may also contribute to the stigmatization of 
unapproved designs or applications. Finally, while all states are 
theoretically equals in the international legal order, in reality some are 
more equal than others. In such an environment, clear rules ensure that 
different states’ actions are evaluated under the same standard. 

The negotiation process necessary to creating international 
regulations can also highlight where the existing law is unclear, 
increasing the likelihood that legal lacunae are prospectively 
addressed.238 Even if states are unable to conclusively determine how to 
fill such gaps, identifying them encourages future discussion, which in 
turn furthers the growth of relevant customary norms. Additionally, the 
regulatory instrument itself can be designed to foster a continued 
conversation, by creating an interpretive body or by requiring state 
parties to convene regularly to consider new issues. 

Stepping back, state interest and engagement in negotiating a treaty 
legitimizes the issue as one deserving of international attention.2° As a 
result, entities and individuals who might not have otherwise considered 
potential impacts of autonomous weapon systems will be more likely to 
do so, and states may be more likely to promulgate national rules and 
policies on the subject. 

Ban advocates argue that a complete ban is preferable to a 
regulatory instrument, as it would be clearer and easier to enforce, 
decrease the possibility that autonomous weapon systems will 


238 This process has already begun. See Chairperson Simon-Michel Report, supra note 6. 
239 Marchant et al., supra note 68, at 314. 
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proliferate and be misused, and would enhance the stigma associated 
with usage by non-state parties.240 These claims are not necessarily true, 
however—as Kenneth Anderson and Matthew Waxman have discussed, 
weapons autonomy will develop incrementally.241 Bans attempt to 
divide white from black; regulation will better address the evolving grey 
zones of weapons autonomy. Additionally, given the low likelihood that 
states will be able to conclude a treaty ban, let alone an effective one, and 
given that a regulatory instrument could provide many of the beneficial 
legal limitations and humanitarian protections associated with a 
complete ban,242 even those who would have preferred a complete ban 
should welcome efforts to create effective regulation. 


C. Options for Intentional International Regulation 


Ideally, autonomous weapon systems would be governed by a 
comprehensive legal regime, comprised of international, transnational, 
and domestic laws. While this ideal may not ever be fully realized, states 
and other parties interested in the governance of this new weaponry can 
begin working toward it now. Indeed, many relevant entities—states, 
developers, manufacturers, programmers, et cetera—would likely 
welcome a clarified ethical and legal framework. 

Accordingly, this subsection discusses possible structures and 
content of a regulatory treaty, as well as other, informal sources of 
guidance and governance. It concludes that an ideal international legal 
regime would consist of a framework convention and a thorough 
collection of associated additional protocols, supplemented by domestic 
law and other sources of informal governance mechanisms. 


1. A Regulatory Treaty 


a. Structural Considerations 


A regulatory treaty on autonomous weapon systems is most likely 
to take one of two forms. It might be negotiated as a sixth additional 
protocol to the Convention on Certain Conventional Weapons (CCW), 
joining the ranks of restrictions on the use of non-detectable fragments, 
mines, incendiary weapons, blinding lasers, and explosive remnants. 
Alternatively, states might negotiate an independent treaty, which 
would ideally take the form of an easily adaptable framework 
convention. 


240 See ADVANCING THE DEBATE, supra note 88, at 22-23. 
241 Anderson & Waxman, supra note 4. 
242 Cf. ADVANCING THE DEBATE, supra note 88, at 3-4. 
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Constructing a regulatory treaty as an additional protocol to the 
CCW is appealing, primarily because there is already a significant 
impetus toward doing so. The first two international meetings of state 
representatives to discuss autonomous weapon system occurred under 
CCW auspices, and such momentum is not to be underestimated. 

That being said, should the CCW process falter or result in 
minimal regulation, state parties interested in constructing a more 
robust legal regime might attempt to negotiate an independent treaty (as 
occurred with landmines and cluster munitions). Furthermore, there is 
some internal inconsistency in drafting regulations for autonomous 
weapon systems as an additional protocol to the CCW: they are hardly 
“conventional” weapons at present, and they are unlikely to be so for 
quite some time. 

An independent treaty might take one of three forms: it might 
attempt comprehensive regulation (like the Chemical Weapons 
Convention—which, in addition to banning the development, 
production, acquisition, stockpiling, retention, transfer, and use of 
certain defined chemical weapons, also outlines enforcement 
mechanisms), provide piecemeal regulations of specific activities (like 
the Nuclear Test Ban or nonproliferation treaties), or serve as a 
framework treaty intended to be augmented by later protocols (like the 
CCW itself). All of these have associated benefits and drawbacks. 

Comprehensive conventions allow states to create an integrated 
and internally reinforcing regulatory regime. However, precisely 
because of its grand substantive scope and varied opportunities for 
disagreement, a comprehensive treaty on autonomous weapon systems 
is unlikely to be concluded and ratified. The Chemical Weapons 
Convention was successful in part because chemical weapons were well 
understood; the possibilities of increasingly autonomous weapon 
systems are as yet unknown. 

Should a comprehensive convention be attempted, drafters will 
likely favor broad, expansive statements over specific rules, both to 
accommodate as-yet unimagined innovations and to encourage higher 
ratification rates. Flexible provisions are certainly to be preferred, as 
specific ones may quickly be rendered obsolete. However, they also are 
more subject to interpretative abuse. Any comprehensive treaty on 
autonomous weapon systems should therefore create or designate an 
authoritative interpretative body with the power to evaluate whether 
new applications accord with the treaty’s overarching object and 
purpose. 

Piecemeal treaties are useful insofar as they allow states to create 
law for specific issues where there is consensus and table the more 
thorny questions that might prevent the widespread ratification of a 
more comprehensive treaty. Additionally, because any given treaty 
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within the regime is focused on a specific issue, it will be able to address 
the subject in greater depth than a more comprehensive treaty—and, in 
the wake of new innovations, it will be more easily amended, as the 
alteration of one treaty will not threaten the regime as a whole. 

What piecemeal treaty regimes gain in particularity and flexibility, 
however, they lose in integration and cohesion. Different treaties in the 
regime will have different objects and aims, which sometimes may 
conflict with each other. Additionally, different treaties will have 
different state parties, so the law for one state will rarely be the law for 
another. 

A framework convention marries many of the benefits of 
comprehensive and piecemeal treaty regimes. The convention itself can 
clarify definitions and guiding principles, while tabling controversial 
issues for a later date. Because ratifying such a convention is non- 
threatening, it is likely to enjoy greater ratification rates than a 
comprehensive treaty and thereby encourage increased state 
involvement in the development of additional protocols. Additional 
protocols can address specific issues and may be amended without 
threatening the legal regime as a whole, but because they are concluded 
under the aegis of the guiding principles of the framework convention, 
there is likely to be more of a focus on integration and internal 
coherence than piecemeal treaty regimes. 

Such an approach is not without problems: the framework 
convention itself is likely to have little to no teeth and on controversial 
subjects—it arguably just kicks the can further down the road, as it is 
only effective to the extent that state parties join the later additional 
protocols resolving such issues.243 Ultimately, however, it is the best 
option. 


b. Subject Matter 


Any initial regulatory treaty will need to include a definition of 
“autonomous weapon system” from a law of armed conflict perspective. 
In constructing such a definition, states will need to determine where to 
set the bar for autonomy, what qualifiers are pertinent and which are 
irrelevant, and whether to except certain currently existing but 
uncontroversial weapon systems. States should also take this 
opportunity to affirm the applicability of the existing law of armed 
conflict principles to the development and use of autonomous weapon 
systems. 244 


243 Id. 
244 Anderson, Reisner & Waxman, supra note 57, at 406-07. 
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After a definition is agreed upon, there are myriad other questions 
for states to consider in evaluating what types and usages of 
autonomous weapon systems should be permissible based on policy, 
ethical, and other considerations. For example, the concept of 
“meaningful human control” has recently emerged as a potential 
guiding principle for the development and use of autonomous weapon 
systems—states might clarify what such control would entail,245 even if 
that standard is difficult to enforce.246 

A regulatory treaty or additional protocol might focus on the 
design or manufacture of autonomous weapon systems, requiring them 
to incorporate components that decrease the possibility of violations of 
the law of armed conflict. One possibility would treat ongoing consent 
as a form of fuel: an autonomous weapon system could be designed to 
check in with a human operator at regular intervals after deployment 
and to shut down or self-destruct if it did not receive reiterated approval 
to continue. Designs that would result in autonomous weapon systems 
with an inappropriate amount of destructive power could also be 
forbidden. 

Additionally or alternatively, a regulatory treaty or additional 
protocol might discuss how autonomous weapon systems may be 
lawfully employed. They might be required to be stationary systems, to 
increase the likelihood that they will only be used for defensive 
purposes. Their potential targets might be restricted to non-human 
entities, or they might be required to be used only for non-lethal 
purposes—perhaps for surveillance, reconnaissance, or search-and- 
rescue missions. Their use might be permitted in certain regions, such 
as outer space and the deep sea, or prohibited in others, such as urban 
or densely populated areas. 

A treaty or additional protocol focused on accountability for law of 
armed conflict violations would have additional requirements. At the 
very least, it should reiterate that states are responsible for the actions of 
their autonomous weapon systems. It might also compel states to adopt 
certain policies and practices regarding the training of human operators 
or the human supervision of autonomous weapon systems. It could also 


245 See Michael C. Horowitz & Paul Scharre, Meaningful Human Control in Weapon Systems: 
A Primer (Mar. 2015) (working paper), available at http://www.cnas.org/sites/default/files/ 
publications-pdf/Ethical_Autonomy_Working_Paper_031315.pdf (suggesting that “meaningful 
human control” has three necessary components: (1) human operators make informed decisions; 
(2) human operators have sufficient information to ensure the lawfulness of their action; and (3) 
the weapon has been designed and tested and the human operator has been trained to ensure 
effective control over the use of the weapon); Sarah Knuckey, Governments Conclude First (Ever) 
Debate on Autonomous Weapons: What Happened and What’s Next, JUST SECURITY (May 16, 
2014, 12:31 PM), http://justsecurity.org/10518/autonomous-weapons-intergovernmental-meeting 
(discussing state support for “meaningful human control” over autonomous weapon systems). 
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mandate that states implement domestic criminal legislation or military 
procedures to establish individual liability for war crimes committed by 
autonomous weapon systems.?47 As noted above, however, autonomous 
weapon systems pose unusual enforcement problems, in that it may be 
impossible for observers to identify when such weaponry is used 
inappropriately. Furthermore, there may be attribution issues: should an 
autonomous weapon system be discovered, independently wreaking 
havoc, it may be impossible to determine what state is responsible for 
deploying it. Any enforcement regime would need to tackle these 
problems. 

While an initial framework treaty will be unlikely to address all 
relevant issues, a comprehensive treaty regime would ideally eventually 
develop and provide guidance regarding the research and development, 
testing, production, sale and transfer, acquisition, and use of 
autonomous weapon systems. 


2. Informal Lawmaking 


Treaties are often the go-to option for proposals for new law, as 
their ability to create binding international legal obligations is 
uncontested. But there simply may not be sufficient state commitment 
or consensus to conclude a regulatory treaty now. There are also some 
persuasive reasons to avoid codifying concrete regulations at present: 
autonomous weapon systems currently in use are not overly 
controversial, and we do not yet have a full understanding of the 
possibilities and risks posed by weapons systems with greater levels of 
autonomy. 48 

But this does not mean we should do nothing. The international 
legal system has myriad alternative sources of guidance and governance, 
many of which can be extremely effective in channeling state action, 
notwithstanding their lack of formal international legal status. These 
sources include common understandings based on international and 
transnational dialogue, nonbinding resolutions and declarations, 
professional guidelines and codes of conduct, civil society reports and 
policy briefs, industry practice, and even domestic laws and policies.249 
Such sources are likely to be both more narrowly tailored and more 


247 See also John Frank Weaver, Asimov’s Three Laws Are Not an International Treaty, SLATE 
(Dec. 1, 2014, 9:54 AM),  http://www.slate.com/articles/technology/future_tense/2014/12/ 
autonomous_weapons_and_international_law_we_need_these_three_treaties_to.html 
(proposing multiple new conventions). 

248 See Marchant et al., supra note 68, at 283-84 (discussing the impossibility of anticipating 
risks inherent in complex new technologies). 

249 See Crootof, supra note 178, at 24. 
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flexible than treaty provisions, and thus better able to address 
unanticipated technological breakthroughs.250 

Two prominent examples of influential, nonformal sources of 
guidance are the Tallinn Manual on the International Law Applicable to 
Cyber Warfare,?5! which was commissioned by the NATO Cooperative 
Cyber Defence Centre of Excellence, and the International Committee 
of the Red Cross’s interpretive guidance regarding the legal status of 
civilians directly participating in hostilities.252 Each of these publications 
was spurred by the fact that new technology—cyberwarfare and drones, 
respectively—raised new questions that were not conclusively addressed 
by the existing law of war. In each case, experts discussed and debated 
how the law should evolve both to satisfy state needs and to preserve 
basic humanitarian protections. Finally, by suggesting rules for 
permissible state action, they sparked wider, ongoing discussions as 
their reasoning and conclusions were evaluated by states, civil society, 
and scholars. States or other interested entities might similarly convene 
a group of experts to issue a comprehensive publication on the law of 
autonomous weapon systems. 

Some of these sources of guidance, like industry practice and 
professional guidelines, will likely develop absent state action. To ensure 
that they do not evolve to undermine fundamental protections of the 
law of armed conflict, states should provide guidance to developers, 
manufacturers, and future operators as to what types and usages of 
autonomous weapon systems are permissible.253 Nor should this be a 
top-down exchange: because these groups will likely have the best 
understanding of actual technical and operational concerns associated 
with autonomous weapon systems, they should be included in any 
attempt to create a comprehensive publication.254 


++ * 


While law is often fated to trail unanticipated technological 
innovations, that need not be the case with autonomous weapon 
systems. Now is the time for states and other interested parties to take 
proactive measures toward the development of an intentional regulatory 


250 See Marchant et al., supra note 68, at 306-13. 
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regime for autonomous weapon systems—one which both permits 
states to explore options for effective weapons and also protects the 
fundamental humanitarian principles of the existing law of armed 
conflict. 


CONCLUSION 


“There are no right answers to the wrong questions. ”255 


The space between proponents and skeptics of a ban on 
autonomous weapon systems is smaller than it may initially appear: all 
parties are interested in preserving the law of armed conflict’s 
humanitarian protections. But those most concerned about the threats 
potentially posed by this new form of weaponry have tended to focus on 
how to ban it, resulting in a spirited debate of the wrong question. 
Claims that autonomous weapon systems could never comply with the 
law of armed conflict and inappropriate comparisons to successful 
weapon bans have succeeded only in further muddying the 
conversation. Meanwhile, states continue to develop and deploy 
autonomous weapon systems in the absence of international 
conversation—much less consensus—on how such weaponry should be 
used. 

In proposing a clarified definition for autonomous weapon 
systems, this Article has attempted to reframe the legal conversation. 
The right question going forward is not how to ban autonomous 
weapon systems, but rather how best to regulate them. We should not 
squander this opportunity to proactively channel the development of 
autonomous weapon systems in futile debates over whether to ban a 
class of uniquely effective weaponry already in widespread use. 


255 URSULA K. LE GUIN, PLANET OF EXILE 1 (1966). 
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APPENDIX: HISTORIC BANS 


Independent books could be (and have been) written on the history 
and circumstances unique to attempted and enacted weapon bans. This 
appendix aims to provide some background on oft-cited bans in the 
autonomous weapon systems debate and the common wisdom 
regarding their successes and failures. 


A. Failed Bans 


Various new technologies have utterly revolutionized warfare, 
often with catastrophic effects for both combatants and civilians. 
Despite sometimes strong and ongoing interest from states and civil 
society, however, attempts to ban certain new weapons have been 
largely unsuccessful. 


1. Crossbows 


Although not a treaty, one of the earliest weapon bans was Pope 
Urban Ils proclamation prohibiting the use of crossbows in 1096—at 
least, in inter-Christian wars.256 At the time, crossbows were seen as 
destabilizing new weaponry that undermined traditional assumptions of 
how warfare was conducted. For the first time in history, a peasant foot 
soldier with little training could easily kill a high-born, professional 
knight257—an “unequivocal violation of the chivalric code.” 25* However, 
due largely to the crossbow’s usefulness, this ban barely outlasted its 
proclamation.259 


2. Aerial Bombardment 


In 1783 the Montgolfier brothers took the first hot air balloon 
flight; nearly a century later, a balloon was first used in an armed 


256 The Crossbow—A Medieval Doomsday Device?, MIL. HIST. Now (May 23, 2012), 
militaryhistorynow.com/2012/05/23/the-crossbow-a-medieval-wmd. Pope Innocent II again 
attempted to ban the crossbow in 1139. Reeves & Johnson, supra note 4, at 27 nn.25-36. 

257 The Crossbow, supra note 256 (noting that “an army of peasants could be made proficient 
with crossbow in weeks, or even days,” as it did not require the strength or skill necessary to 
operate a bow). 

258 Reeves & Johnson, supra note 4, at 27. 

259 W.T. Mallison, Jr., The Laws of War and the Juridical Control of Weapons of Mass 
Destruction in General and Limited Wars, 36 GEO. WASH. L. REV. 308, 316 (1967); Reeves & 
Johnson, supra note 4, at 27. 
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conflict for military reconnaissance.2© In recognition of the possibility 
of balloon-launched bombs, and their likely inaccurate and thus 
indiscriminate nature, states discussed a potential ban on aerial 
bombardment at the 1899 Hague Peace Conference.?+! Ultimately, 
twenty-four states—including powerhouses like Austria-Hungary, 
China, France, Germany, Italy, Japan, Russia, and Spain2°2—ratified a 
Declaration prohibiting, for five years, “the launching of projectiles and 
explosives from balloons, or by other new methods of a similar 
nature.” 263 

The attempt to renew the ban after the Wright Brother’s 1903 
flight, however, was less successful. At the 1907 Hague Peace 
Conference, a Declaration with an identical prohibition was drafted, 
again as a temporary ban intended to last only until the third Peace 
Conference.?6+ France, Italy, Japan, Spain, and Russia did not ratify the 
new Declaration; Austria-Hungary signed but never ratified; and 
Germany’s ratification was conditional on all other participating parties’ 
ratification.265 Arthur Kuhn, writing in 1910, suggested that the 
Declaration’s relative lack of success was likely the result of states’ 
interest in exploring the military applications of the new technology.26 

Recognizing the lack of specific regulations governing air warfare, 
states appointed a Commission of Jurists to develop rules.267 The 
result—the 1923 Hague Rules of Aerial Warfare2ss—prohibited aerial 
bombardment for “the purpose of terrorizing the civil population or 
destroying or damaging private property without military character or 
injuring non-combatants” and clarified that aerial bombardment was 
lawful only if directed at a military objective.269 Although the Rules were 
never ratified, they were recognized as clarifying customary 
international law and regularly cited by the League of Nations.2”° 


260 Javier Guisandex Gómez, The Law of Air Warfare, 323 INT'L REV. RED CROSS 347, 348 
(1998). 

261 Id. at 349-50. 

262 The United States signed, but did not ratify, the 1899 Declaration. 

263 Declaration (IV, I), to Prohibit, for the Term of Five Years, the Launching of Projectiles and 
Explosives from Balloons, and Other Methods of a Similar Nature, July 29, 1899, 32 Stat. 1839, 1 
Bevans 270. 

264 1907 Declaration, supra note 211. 

265 See Matthew Lippman, Aerial Attacks on Civilians and the Humanitarian Law of War: 
Technology and Terror from World War I to Afghanistan, 33 CAL. W. INT'L L.J. 1, 6 (2002). 

266 Arthur K. Kuhn, The Beginnings of an Aérial Law, 4 AM. J. INTL L. 109, 119-20 (1910). 

267 Gómez, supra note 260, at 352. 

268 Rules Concerning the Control of Wireless Telegraphy in Time of War and Air Warfare, 
Dec. 1922-Feb. 1923 (drafted by a Commission of Jurists at the Hague). 

269 Id. arts. 22, 24. Other articles protected certain civilian buildings, including religious, 
medical, artistic, scientific, and historic structures. Id. arts. 25-26. 

270 Lippman, supra note 265, at 11, 12-14. 
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At the beginning of World War II, the parties attempted to avoid 
using aerial bombardment in urban areas.27! This ended with the 
September 1939 German aerial attack on Warsaw; in response, Great 
Britain (and, eventually, the United States) began targeting civilian 
population centers as part of their overall aerial strategic plan.272 The 
result was devastating: relentless and indiscriminate strikes by both sides 
led to unprecedented numbers of civilian deaths273—and, ultimately, to 
the bombing of Hiroshima and Nagasaki. 

Today, aerial bombardment is governed by a patchwork of treaties 
relevant to air warfare and by the law of armed conflict generally.274 


3. Submarines 


Although military submarines were never formally prohibited, 
states concluded agreements which so strictly limited their usage that 
they amounted to bans. However, these proscriptions proved largely 
unenforceable in practice. 

As an island nation and naval power, Great Britain sensed its 
vulnerability to submarine warfare.2’5 At the Hague Peace Conference of 
1899, with Germany’s support, it tried and failed to outlaw submarines 
as a weapon of war.276 

After Germany used submarines with devastating effect in World 
War I, during the 1923 Washington Conference Great Britain again 
tried, and again failed, to completely ban the use of submarines.277 
However, based on a U.S. proposal, the Conference participants adopted 
a general resolution that prohibited the destruction of a merchant vessel 
“unless its crew and passengers have been placed in safety.”278 This 
customary rule, originally applicable to surface warships, was explicitly 
made applicable to submarines in two later treaties. The 1930 London 
Naval Treaty prohibited submarines from neutralizing potentially 
hostile merchant vessels without having first ensured the safety of their 


271 See id. at 15. 

272 Id. at 15-16 & n.143. 

273 Id. at 15-19. 

274 See, e.g., Protocol on Prohibitions or Restrictions on the Use of Incendiary Weapons 
(Protocol III) art. 2(2), Oct. 10, 1980, S. Treaty Doc. No. 105-1, 1342 U.N.T.S. 171. 

275 See D.P. O'Connell, International Law and Contemporary Naval Operations, 44 BRIT. Y.B. 
INT'L L. 19, 45 (1970). 

276 Id. 

277 Id. at 49. 
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passengers, crew, and ships papers,?79 prohibitions which were 
reiterated in the 1936 London Protocol.280 

But not only was it impossible for Great Britain to ban submarine 
warfare, the treaties delineating how submarines might be lawfully used 
proved almost immediately unenforceable. Submarines depend on 
stealth: they cannot escort captured vessels to ports to ensure the safety 
of their crew and passengers. Nor, due to space constraints, can they 
take additional individuals on board, much less potentially hostile ones. 
As a result, despite the fact that nearly all major naval powers acceded to 
the Protocol,?8! during World War II all states with submarines (save 
Japan) engaged in some form of unrestricted submarine warfare in 
violation of their treaty obligations.282 As one scholar noted, the 
Protocol’s requirement was “an unworkable ideal couched in ambiguous 
terms which did not address the practicalities of submarine warfare.”283 

Although the London Protocol still technically states the treaty law 
relevant to submarines,” in reality their military usage is governed 
largely by the law of armed conflict generally and customary 
international law specific to submarines. 285 


4. Nuclear Weapons 


On August 6, 1945, the United States detonated a uranium bomb 
over Hiroshima, killing more than 140,000 people within a few 
months.28 On August 9, it detonated a plutonium bomb over Nagasaki, 
resulting in the deaths of approximately 74,000 people by the close of 
that year.287 Five months later, in its first resolution, the U.N. General 
Assembly called for the complete elimination of nuclear weapons.288 
Over the next few decades, states and civil society attempted to address 
the nuclear weapon threat through a variety of means. 
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In 1994, the General Assembly requested that the International 
Court of Justice issue an advisory opinion in response to the question: 
“Is the threat or use of nuclear weapons in any circumstance permitted 
under international law?”289 After extensive arguments and much 
deliberation, a split court concluded that no treaty or customary 
international law prohibited the use of nuclear weapons, but 


the threat or use of nuclear weapons would generally be contrary to 
the rules of international law applicable in armed conflict, and in 
particular the principles and rules of humanitarian law; 


However, in view of the current state of international law, and of the 
elements of fact at its disposal, the Court cannot conclude 
definitively whether the threat or use of nuclear weapons would be 
lawful or unlawful in an extreme circumstance of self-defence, in 
which the very survival of a State would be at stake . . . .290 


In other words, the court could not find that nuclear weapons were 
per se unlawful, as a situation could be envisioned in which they could 
be lawfully used. Such use remains regulated by the law of armed 
conflict, however, and there remains “the possibility that such a weapon 
could be unlawful by reference to the humanitarian law, if its use could 
never comply with its requirements.”29! In the absence of an official 
pronouncement of per se illegality, the only hope for a complete ban on 
nuclear weapons is through state action—and states have proven 
unwilling to enact such a ban. 

The 1968 Non-Proliferation Treaty does not ban nuclear weapons 
outright, but rather aims to limit their proliferation and eventually 
achieve universal disarmament.?»2 The crux of a treaty is an agreement 
between states with and without nuclear weapons: the latter vow not to 
acquire nuclear weapon technology, and the former agree to pursue 
disarmament and to share the benefits of non-military nuclear 
technology. The 190 states party to the Treaty—more than any other 
arms limitation or disarmament agreement—include the five original 
nuclear weapon states (China, France, Russia, the United Kingdom, and 


289 Legality of the Threat or Use of Nuclear Weapons (Request for an Advisory Opinion), G.A. 
Res. 49/75(K), U.N. GAOR, 49th Sess., Supp. No. 49, U.N. Doc. A/RES/49/75, at 6 (Dec. 15, 1994). 

290 Nuclear Weapons, supra note 167, at 266. However, President Bedjaoui cautioned “that the 
Court’s inability to go beyond this statement of the situation can in no way be interpreted to mean 
that it is leaving the door ajar to recognition of the legality of the threat or use of nuclear 
weapons.” Id. at 270 (declaration of President Bedjaoui). 

291 Id. at 589 (dissenting opinion of Judge Higgins); see also id. at 320 (dissenting opinion of 
Vice-President Schwebel) (“It cannot be accepted that the use of nuclear weapons on a scale 
which would—or could—result in the deaths of many millions in indiscriminate inferno and by 
far-reaching fallout, have profoundly pernicious effects in space and time, and render 
uninhabitable much or all of the earth, could be lawful.”). 

292 Treaty on the Non-Proliferation of Nuclear Weapons, July 1, 1968, 21 U.S.T. 483, 729 
U.N.T.S. 161 (entered into force Mar. 5, 1970). 
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the United States).293 However, of the five states not party to the treaty, 
four—India, Israel, North Korea, and Pakistan—are the only other states 
known or suspected of having nuclear weapons. The Treaty has thus 
been critiqued on the grounds that only slightly over half of the total 
number of states with nuclear weapons are state parties. On the other 
hand, the fact that only nine states currently have nuclear weapons 
might be an indicator of the Treaty’s success. The Treaty has also been 
criticized as benefiting nuclear weapon states at the expense of non- 
nuclear states, as the former have yet to eliminate their nuclear 
stockpiles. 

The testing of nuclear weapons is also regulated by two treaties. In 
1963, the United States, United Kingdom, and Soviet Union concluded 
the Partial Test Ban Treaty, which banned nuclear tests in the 
atmosphere, outer space, or underwater.2%4 Although an important step 
forward, the Treaty did not prohibit underground testing. Additionally, 
neither China nor France became state parties at the time, and North 
Korea never acceded. In 1996, states attempted to close the 
underground loophole by concluding the Comprehensive Nuclear-Test- 
Ban Treaty, which prohibits nuclear explosions in all environments, for 
military or for civilian purposes.295 There are currently 163 state parties 
to the Treaty.2°* However, it cannot enter into force until after all forty- 
four listed “nuclear weapon possible” states ratify it; at present, eight 
have not done so.?9 China, Egypt, Iran, Israel, and the United States 
have signed but not ratified the Treaty; India, North Korea, and 
Pakistan have not signed it.29 Since the Treaty opened for signature, 
India, North Korea, and Pakistan have all tested nuclear weapons.?9 

Currently, nuclear weapons are governed by the various specific 
treaties regulating their use and by the law of armed conflict generally. 300 


293 Treaty on the Non-Proliferation of Nuclear Weapons (NPT), UN OFF. FOR DISARMAMENT 
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B. Un/Successful Bans 


It is easy to determine that an attempted weapon ban has failed: 
either states are unable to conclude a treaty, or no one can credibly 
argue that a purported ban actually keeps states from employing a 
weapon. It is less clear when a ban has been successful. Is success 
measured by the number of state parties? By the number of relevant 
state parties—which is to say, the primary producers and users of a 
weapon or by the major military powers? By state party compliance? By 
the relative level of state party compliance when compared with other 
treaties regarding conduct in warfare?30! By the number of times the 
weapon in question has been used by any state since the ban was 
enacted? Or by whether the use of the weapon has become so 
stigmatized that the ban is recognized as customary international law? 

This subsection discusses commonly cited “successful” bans— 
defined here as bans which are both enacted and effective at limiting the 
usage of the banned weapon—organized roughly from the least to most 
obviously effective. Again, a thorough analysis of each ban is beyond the 
scope of this Article, but these brief descriptions do highlight certain 
commonalities. 


1. Cluster Munitions 


Cluster munitions are weapons that disperse or release 
submunitions or “bomblets.” These submunitions tend to be small, 
unguided explosive devices, designed to detonate just prior to, on, or 
after impact. Submunitions may not deploy when or as intended, 
however, leaving unexploded ordnance that can harm civilians long 
after the armed conflict has concluded. Protocol V to the Convention on 
Certain Conventional Weapons aimed to minimize their impact by, 
among other things, obligating member states to remove or destroy 
unexploded ordnance in their territories when feasible and to take 
“generic preventive measures aimed at minimi[z]ing the occurrence of 
explosive remnants of war.” 302 

The 2008 Convention on Cluster Munitions prohibits state parties 
from using, developing, producing, acquiring, stockpiling, retaining, or 


disarmament/WMD/Nuclear/NWFZ.shtml (last visited Apr. 27, 2015) (with links to relevant 
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transferring cluster munitions under any circumstances.33 It further 
requires state parties to clear unexploded ordnance and to destroy 
existing stockpiles of such weapons, save only for those to be used for 
research and training related to detection, clearance, and destruction 
techniques.3°4 The Convention currently has eighty-nine state parties, 
which do not include major military powers (China, Russia, or the 
United States) or many states which have recently used cluster 
munitions (Eritrea, Ethiopia, Israel, Georgia, Libya, Russia, Thailand, 
Syria, and the United States).305 

The success of this ban has yet to be determined. Skeptics of its 
success suggest that it carries little power, as many major cluster 
munition producers, stockpilers, and users have not ratified it.306 
Advocates point to the facts that “[a]t least 33 countries that have 
stockpiled, produced, and/or used cluster munitions have signed” and 
that “[a]lmost half the world, including states from every region, has 
signed the Convention” to argue that there is now “widespread 
international rejection of cluster munitions.” 307 


2. Anti-Personnel Landmines 


Anti-personnel landmines are explosive devices designed to be 
triggered by human beings (and are to be distinguished from landmines 
generally, which include anti-tank or anti-vehicle landmines). Their use 
was first regulated by the since-amended Protocol II to the Convention 
on Certain Conventional Weapons, which prohibits the use of mines 
against the civilian population and set forth various requirements 
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constraining their use, placement, advertisement, and neutralization. 308 
One hundred and two states are party to the Amended Protocol.309 

The adoption of the 1997 Mine Ban Convention, also called the 
Ottawa Treaty, “marked the first time in the history of international 
humanitarian law that States agreed to ban a weapon that was in 
widespread use throughout the world.”3!0 The Convention prohibits the 
use, development, production, acquisition, stockpiling, and direct or 
indirect transfer of anti-personnel landmines (save for destruction or 
training in detection, clearance, and destruction).3!! One hundred and 
sixty-one states are party to the Convention, although these do not 
include China, Russia, or the United States. 312 

Since the Convention took effect, there has been a dramatic 
reduction in the production and use of anti-personnel landmines.33 
However, states are still using these landmines, and it is generally agreed 
that the ban has not (yet) evolved into customary international law.314 


3. Biological Weapons 


Biological weapons include both harmful biological agents and the 
means of their delivery.315 Although its primary aim was forbidding the 
use of poisonous gases in warfare, the 1925 Geneva Gas Protocol also 
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“extend[ed] this prohibition to the use of bacteriological methods of 
warfare.”316 

This initial ban was strengthened by the 1972 Biological Weapons 
Convention, the first multilateral disarmament treaty banning the 
creation of an entire class of weaponry.3!7 The Convention does not 
explicitly ban the use of biological weapons—only their development, 
production, and stockpiling. A prohibition on their use is an obviously 
implicit goal, however, and the Convention does note that it is not 
intended to limit states’ obligations under the Geneva Gas Protocol, 
which does explicitly prohibit the use of biological agents in warfare.318 
There are 171 state parties to the Convention; an additional nine have 
signed but not yet acceded, and sixteen states are not members.3!9 

The success of the Biological Weapons Convention is debatable. 
While it certainly serves as an example of states’ ability to conclude a 
ban regarding an entire class of weaponry, states’ failure to explicitly ban 
the use of biological weapons and the lack of enforcement mechanisms 
seem to render it toothless. 

Nonetheless, the ICRC has concluded that the prohibition on the 
use of biological weapons in international and non-international armed 
conflicts is customary international law.320 This is a credible conclusion, 
given the infrequency of violations, that “[v]irtually all allegations of 
possession by States have been denied,” and the near-universal 
condemnation of Iraq (then not a member state to the Convention) 
when it was discovered in the mid-1990s that it was pursuing biological 
weapons research. 32! 


4. Chemical Weapons 


Chemical weapons include both toxic chemicals and the munitions 
or devices designed to disperse them. Notwithstanding initial 
prohibitions against poisons and asphyxiating gases resulting from the 
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1899 and 1907 Hague Peace Conferences,322 in World War I both sides 
used chemical weapons extensively. 323 

Largely in reaction to the horrors of the gas offensive, states 
concluded the 1925 Geneva Gas Protocol. It states that “the use in war 
of asphyxiating, poisonous or other gases, and of all analogous liquids, 
materials or devices, has been justly condemned by the general opinion 
of the civilized world,” and that the state parties therefore accept the 
prohibition on their use—at least against other member states.324 There 
are currently 138 state parties to the Protocol.325 And, in World War II, 
neither side made significant use of chemical weapons. 326 

In response to various incidents of chemical weapon usage during 
the mid-twentieth century, states concluded the comprehensive 1992 
Chemical Weapons Convention. It prohibits the development, 
production, acquisition, stockpiling, retention, transfer, and use of 
chemical weapons.327 Among other enforcement mechanisms, it 
requires state parties to submit to inspections upon the challenge of any 
other state party.328 One hundred and ninety states are currently party to 
the Convention.329 

Based on treaty law and state practice, the ICRC has determined 
that the prohibition on the use of chemical weapons in international or 
non-international armed conflicts is now customary international 
law.330 This conclusion is further bolstered by the Rome Statute’s 
prohibition on the use of chemical weapons in international armed 
conflicts,331 the limited number of Convention violations, and states’ 
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329 Convention on the Prohibition of the Development, Production, Stockpiling and Use of 
Chemical Weapons and on Their Destruction, UN TREATY COLLECTION, https://treaties.un.org/ 
pages/ViewDetails.aspx?src=TREATY &mtdsg_no=XXVI-3&chapter=26&lang=en (last visited 
Apr. 27, 2015). Israel and Myanmar have signed but not yet ratified the Convention; Angola, 
Egypt, North Korea, and South Sudan have neither signed nor ratified. Id. 

330 Rule 74. Chemical Weapons, INT'L COMMITTEE RED CROSS, http://www.icrc.org/customary- 
ihl/eng/docs/v1_cha_chapter24_rule74 (last visited Apr. 27, 2015). 

331 Rome Statute of the International Criminal Court art. 8(2)(b)(xviii), July 17, 1998, 2187 
UN.T.S. 90 (banning the use of “asphyxiating, poisonous or other gases, and all analogous 
liquids, materials or devices”). 
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overwhelmingly negative reaction to Syria’s 2013 use of chemical 
weapons (prior to its September 2013 accession to the Convention). 


5. Permanently Blinding Lasers 


The 1995 Protocol IV to the Convention on Certain Conventional 
Weapons prohibits state parties from employing or transferring “laser 
weapons specifically designed, as their sole combat function or as one of 
their combat functions, to cause permanent blindness to unenhanced 
vision.”332 There are currently 104 state parties to the Protocol,333 and 
the ICRC has found that its prohibition has attained the status of 
customary international law in both international and non-international 
armed conflicts.334 

The permanently blinding laser ban has two notable characteristics. 
First, it is one of the few prospective bans: although in development at 
the time of the Protocol’s conclusion, blinding lasers had not yet been 
deployed.335 Second, it may be the most successful ban of all time. Not 
only is there little controversy over what weapons it forbids and 
permits,336 there are no recorded violations. 


332 Protocol IV, supra note 207, art. 1. 

333 Additional Protocol to the Convention on Prohibitions or Restrictions on the Use of Certain 
Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have 
Indiscriminate Effects (Protocol IV, entitled Protocol on Blinding Laser Weapons), UN TREATY 
COLLECTION, https://treaties.un.org/Pages/ViewDetails.aspx?src=TREATY &mtdsg_no=XXVI-2- 
a&chapter=26&lang=en (last visited Apr. 27, 2015). 

334 Rule 86. Blinding Laser Weapons, INT'L COMMITTEE RED CROSS, http://www.icrc.org/ 
customary-ihl/eng/docs/v1_rul_rule86 (last visited Apr. 27, 2015). 

335 Another, less frequently mentioned prospective ban was the 1899 prohibition on the use of 
projectiles intended to diffuse asphyxiating gases. See PRICE, supra note 202, at 15-16. 

336 This is in stark contrast to other, “successful” bans, such as the prohibition on the use of 
expanding bullets. Although this ban is technically adhered to by all states, there is significant 
controversy regarding what modern weapons it encompasses. See Robin Coupland & Dominique 
Love, The 1899 Hague Declaration Concerning Expanding Bullets: A Treaty Effective for More than 
100 Years Faces Complex Contemporary Issues, 85 INT'L REV. RED CROSS 135, 136 (2003). 
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Joy Buolamwini presenting her research at a TED conference. PHOTO VIA TED 


Joy Buolamwini was conducting research at MIT on how computers recognized people’s faces, when she 


started experiencing something weird. 


Whenever she sat before a system's front-facing camera, it wouldn't recognize her face, even after 
working for her lighter-skinned friends. But when she put on a simple white mask, the face-tracking 


animation suddenly lit up the screen. 
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Suspecting a more widespread problem, she carried out a study on the AI-powered facial recognition 
systems of Microsoft, IBM and Face++, a Chinese startup that has raised more than $500 million from 


investors. 
Buolamwini showed the systems 1,000 faces, and told them to identify each as male or female. 
All three companies did spectacularly when discerning between white faces, and men in particular. 


But when it came to dark-skinned females, the results were dismal: there were 34% more errors with 
dark-skinned females than light-skinned males, according to the findings Buolamwini 

presented on Saturday, Feb. 24th, at the Conference on Fairness, Accountability and Transparency in 
New York. 


As skin shades on women got darker, the chances of the algorithms predicting their gender accurately 
“came close to a coin toss.” With the darkest skin women, the face-detection systems were getting their 


gender wrong close to half the time. 
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Buolamwini’s project, which became the basis of her MIT thesis, shows that concerns about bias 


are adding a new dimension to the general anxiety around artificial intelligence. 


While much has been written about ways that machine learning will replace human jobs, the public has 


paid less attention to the consequences of biased datasets. 


What happens, for instance, when software engineers train their facial-recognition algorithms primarily 


with images of white males? Buolamwini's research showed the algorithm itself becomes prejudiced. 


Another example came to light in 2016, when Microsoft released its AI chatbot Tay onto Twitter. 
Engineers programmed the bot to learn human behavior by interacting with other Twitter users. After 


just 16 hours, Tay was shut down because its tweets had become a stream of sexist, pro-Hitler messages. 
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Experts later said Microsoft had done just fine teaching Tay to mimic behavior, but not enough 


about what behavior was appropriate. 


Suranga Chandratillake, a leading venture capitalist with Balderton Capital in London, UK, says bias in 


AI is as much a concerning issue as that of job destruction. 


“Tm not negative about the job impact,” he says. The bigger issue is building Al-powered systems that 


take historical data, then use it to make judgements. 


“Historical data could be full of things like bias,” Chandratillake says from his office in Kings Cross, which 


is just up the road from the headquarters of Google’s leading artificial intelligence business, DeepMind. 


“On average people approve mortgages to men or people who are white, or from a certain town.” When 


the power to make that judgement is given to a machine, the machine “encodes that bias.” 


So far the examples of bias caused by algorithms have seemed mundane, but in aggregate they can have 
an impact, especially with so many companies racing to incorporate AI into their apps and services. 
(Mentions of "AI" in earnings calls have skyrocketed over the past year, according to CB Insights, even 


from unlikely companies like Procter & Gamble or Bed Bath & Beyond.) 


In recent months several researchers have pointed to how even Google Translate has shown signs of 
sexism, automatically suggesting words like “he” for male-dominated jobs and vice versa, when 


translating from a gender-neutral language like Turkish. 


Camelia Boban, a software developer in Italy, also noticed on Feb. 4th that Google Translate didn’t 
recognize the female term for “programmer” in Italian, which is programmatrice. (She said in a recent 


email to Forbes that the issue has since been corrected.) 
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Such examples might sound surprising when you expect software to be logical and objective. “People 
believe in machines being rational," Chandratillake says. "You end up not realizing that actually, what 


should be meritocratic, isn’t at all. It’s just an encoding of something that wasn’t in the first place.” 


When humans make important decisions about hiring, or granting a bank loan, they’re more likely to be 
questioned about their judgement. There's less reason to question AI because of “this veneer of 


innovative new tech," he says. "But it’s destined to repeat the errors of the past.” 


Today's engineers are also overly-focused on building algorithms to solve complex problems, rather 
than building an algorithm to monitor and report on how the first algorithm is performing -- a kind of 


algorithmic watchdog. 


“Today the way a lot of AI is configured, is basically as a black box,” he adds. “Neural networks are not 


good at explaining why they made a decision.” 
MIT’s Buolamwini points to the lack of diversity in images and data used to train algorithms. 
Fortunately, this is an issue that can be worked on. 


After MIT's Buolamwini sent the results of her study to Microsoft, IBM and Face++, IBM responded by 
replicating her research internally, and releasing a new API, according to a conference goer who attended 


her presentation on Saturday. 


The updated system now classifies darker-skinned females with a success rate of 96.5%. 


Follow me on Twitter @parmy or email me here. 
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I cover developments in AI, robotics, chatbots, digital assistants and emerging tech in Europe. I've 
spent close to a decade profiling the hackers and dreamers who are bringing the most cutting-edge 


technology into our lives, for better or worse. I'm the author of "We Are An... MORE 
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THE SCORED SOCIETY: DUE PROCESS FOR 
AUTOMATED PREDICTIONS 


Danielle Keats Citron” & Frank Pasquale” 


Abstract: Big Data is increasingly mined to rank and rate individuals. Predictive 
algorithms assess whether we are good credit risks, desirable employees, reliable tenants, 
valuable customers—or deadbeats, shirkers, menaces, and “wastes of time.” Crucial 
opportunities are on the line, including the ability to obtain loans, work, housing, and 
insurance. Though automated scoring is pervasive and consequential, it is also opaque and 
lacking oversight. In one area where regulation does prevail—credit—the law focuses on 
credit history, not the derivation of scores from data. 


Procedural regularity is essential for those stigmatized by “artificially intelligent” scoring 
systems. The American due process tradition should inform basic safeguards. Regulators 
should be able to test scoring systems to ensure their fairness and accuracy. Individuals 
should be granted meaningful opportunities to challenge adverse decisions based on scores 
miscategorizing them. Without such protections in place, systems could launder biased and 
arbitrary data into powerfully stigmatizing scores. 
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[Jennifer is] ranked 1,396 out of 179,827 high school students 
in Iowa... . Jennifer’s score is the result of comparing her test 
results, her class rank, her school’s relative academic strength, 
and a number of other factors. ... 
[C]an this be compared against all the other students in the 
country, and maybe even the world? ... 
That’s the idea.... 
That sounds very helpful. . . . And would eliminate a lot of doubt 
and stress out there. 

—Dave Eggers, The Circle’ 


INTRODUCTION TO THE SCORED SOCIETY 


In his novel The Circle, Dave Eggers imagines persistent surveillance 
technologies that score people in every imaginable way. Employees 
receive rankings for their participation in social media.” Retinal apps 
allow police officers to see career criminals in distinct colors—yellow 
for low-level offenders, orange for slightly more dangerous, but still 
nonviolent offenders, and red for the truly violent.’ Intelligence agencies 
can create a web of all of a suspect’s contacts so that criminals’ 
associates are tagged in the same color scheme as the criminals 
themselves.“ 

Eggers’s imagination is not far from current practices. Although 
predictive algorithms may not yet be ranking high school students 
nationwide, or tagging criminals’ associates with color-coded risk 
assessments, they are increasingly rating people in countless aspects of 
their lives. 

Consider these examples. Job candidates are ranked by what their 
online activities say about their creativity and leadership.” Software 
engineers are assessed for their contributions to open source projects, 





. DAVE EGGERS, THE CIRCLE 340-41 (2013) (internal quotation marks omitted). 

. Id. at 190. 

. Id. at 419-20. 

. Id. at 420. 

. See Don Peck, They’re Watching You at Work, ATLANTIC MONTHLY, Dec. 2013, at 72, 76. 
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with points awarded when others use their code. Individuals are 
assessed as likely to vote for a candidate based on their cable-usage 
patterns.’ Recently released prisoners are scored on their likelihood of 
recidivism." 

How are these scores developed? Predictive algorithms mine personal 
information to make guesses about individuals’ likely actions and risks.’ 
A person’s on- and offline activities are turned into scores that rate them 
above or below others.'° Private and public entities rely on predictive 
algorithmic assessments to make important decisions about 
individuals." 

Sometimes, individuals can score the scorers, so to speak. Landlords 
can report bad tenants to data brokers while tenants can check abusive 
landlords on sites like ApartmentRatings.com. On sites like Rate My 
Professors, students can score professors who can respond to critiques 
via video. In many online communities, commenters can in turn rank the 
interplay between the rated, the raters, and the raters of the rated, in an 
effort to make sense of it all (or at least award the most convincing or 
popular with points or “karma”).'” 

Although mutual-scoring opportunities among formally equal subjects 
exist in some communities, the realm of management and business more 
often features powerful entities who turn individuals into ranked and 
rated objects. While scorers often characterize their work as an oasis of 


6. See E. GABRIELLA COLEMAN, CODING FREEDOM 116-22 (2013) (exploring Debian open 
source community and assessment of community members’ contributions). 


7. See Alice E. Marwick, How Your Data Are Being Deeply Mined, N.Y. REV. BOOKS, Jan. 9, 
2014, at 22, 22. 


8. Danielle Keats Citron, Data Mining for Juvenile Offenders, CONCURRING OPINIONS (Apr. 21, 
2010, 3:56 PM), http://www.concurringopinions.com/archives/2010/04/data-mining-for-juvenile- 
offenders.html. 

9. Frank Pasquale, Restoring Transparency to Automated Authority, 9 J. ON TELECOMM. & HIGH 
TECH. L. 235, 235-36 (2011). 


10. Hussein A. Abdou & John Pointon, Credit Scoring, Statistical Techniques and Evaluation 
Criteria: A Review of the Literature, 18 INTELLIGENT SYSTEMS ACCT. FIN. & MGMT. 59, 60-61 
(2011). 

11. See Marwick, supra note 7, at 24; see also Jack Nicas, How Airlines Are Mining Personal 
Data In-Flight, WALL ST. J., Nov. 8, 2013, at B1. 

12. Oren Bracha & Frank Pasquale, Federal Search Commission? Access, Fairness, and 
Accountability in the Law of Search, 93 CORNELL L. REV. 1149, 1159 (2008) (“This structures [sic] 
results in a bottom-up filtration system. At the lowest level, a large number of speakers receive 
relatively broad exposure within local communities likely composed of individuals with high- 
intensity interest or expertise. Speakers who gain salience at the lower levels may gradually gain 
recognition in higher-order clusters and eventually reach general visibility.” (footnotes omitted)). 


13. See JARON LANIER, WHO OWNS THE FUTURE? 108 (2014); JARON LANIER, YOU ARE NOT A 
GADGET (2010). For the distinction between management and community, see generally ROBERT 
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opportunity for the hardworking, the following are examples of ranking 
systems that are used to individuals’ detriment. A credit card company 
uses behavioral-scoring algorithms to rate consumers’ credit risk 
because they used their cards to pay for marriage counseling, therapy, or 
tire-repair services.'* Automated systems rank candidates’ talents by 
looking at how others rate their online contributions.” Threat 
assessments result in arrests or the inability to fly even though they are 
based on erroneous information.'® Political activists are designated as 
“likely” to commit crimes.” 

And there is far more to come. Algorithmic predictions about health 
risks, based on information that individuals share with mobile apps 
about their caloric intake, may soon result in higher insurance 
premiums.'® Sites soliciting feedback on “bad drivers” may aggregate 
the information, and could possibly share it with insurance companies 
who score the risk potential of insured individuals. ° 

The scoring trend is often touted as good news. Advocates applaud 
the removal of human beings and their flaws from the assessment 
process. Automated systems are claimed to rate all individuals in the 
same way, thus averting discrimination. But this account is misleading. 
Because human beings program predictive algorithms, their biases and 
values are embedded into the software’s instructions, known as the 
source code and predictive algorithms.” Scoring systems mine datasets 
containing inaccurate and biased information provided by people.” 
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14. Complaint for Permanent Injunction and Other Equitable Relief at 35, FTC v. CompuCredit 
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Intelligence Apparatus, 62 HASTINGS L.J. 1441, 1444-45 (2011); David Gray & Danielle Keats 
Citron, The Right to Quantitative Privacy, 98 MINN. L. REV. 62, 81 (2013). 

17. See S. PERMANENT SUBCOMM. ON INVESTIGATIONS, 112TH CONG., FEDERAL SUPPORT FOR 
AND INVOLVEMENT IN STATE AND LOCAL FUSION CENTERS 104-05 (2012), available at 
https://www.hsdl.org/?view&did=723145; Danielle Keats Citron & David Gray, Addressing the 
Harm of Total Surveillance: A Reply to Professor Neil Richards, 126 HARV. L. REV. F. 262, 266 
(2013). 

18. See Marwick, supra note 7, at 24. 

19. See Frank Pasquale, Welcome to the Panopticon, CONCURRING OPINIONS (Jan. 2, 2007), 
http://www.concurringopinions.com/archives/2007/01/welcome_to_the_14.html. 

20. Danielle Keats Citron, Technological Due Process, 85 WASH. U. L. REV. 1249, 1260-63 
(2008). 

21. Id.; Danielle Keats Citron, Open Code Governance, 2008 U. CHI. LEGAL F. 355, 363-68 
[hereinafter Citron, Open Code Governance]. 
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There is nothing unbiased about scoring systems. 

Supporters of scoring systems insist that we can trust algorithms to 
adjust themselves for greater accuracy. In the case of credit scoring, 
lenders combine the traditional three-digit credit scores with “credit 
analytics,” which track consumers’ transactions. Suppose credit- 
analytics systems predict that efforts to save money correlates with 
financial distress. Buying generic products instead of branded ones could 
then result in a hike in interest rates. But, the story goes, if consumers 
who bought generic brands also purchased items suggesting their 
financial strength, then all of their purchases would factor into their 
score, keeping them from being penalized from any particular purchase. 

Does everything work out in a wash because information is seen in its 
totality? We cannot rigorously test this claim because scoring systems 
are shrouded in secrecy. Although some scores, such as credit, are 
available to the public, the scorers refuse to reveal the method and logic 
of their predictive systems.” No one can challenge the process of 
scoring and the results because the algorithms are zealously guarded 
trade secrets.” As this Article explores, the outputs of credit-scoring 
systems undermine supporters’ claims. Credit scores are plagued by 
arbitrary results. They may also have a disparate impact on historically 
subordinated groups. 

Just as concerns about scoring systems are more acute, their human 
element is diminishing. Although software engineers initially identify 
the correlations and inferences programmed into algorithms, Big Data 
promises to eliminate the human “middleman” at some point in the 
process.” Once data-mining programs have a range of correlations and 
inferences, they use them to project new forms of learning. The results 
of prior rounds of data mining can lead to unexpected correlations in 
click-through activity. If, for instance, predictive algorithms determine 
not only the types of behavior suggesting loan repayment, but also 
automate the process of learning which adjustments worked best in the 
past, the computing process reaches a third level of sophistication: 
determining which metrics for measuring past predictive algorithms 
were effective, and recommending further iterations for testing.” In 





22. Tal Zarsky, Transparent Predictions, 2013 ILL. L. REV. 1503, 1512. 


23. Evan Hendricks, Credit Reports, Credit Checks, Credit Scores, A.B.A. GPSOLO, July/Aug. 
2011, at 32, 34. 


24. Chris Anderson, The End of Theory: The Data Deluge Makes Scientific Inquiry Obsolete, 
WIRED (June 23, 2008), http://www.wired.com/science/discoveries/magazine/16-07/pb_theory. 


25. A pioneer of artificial intelligence described this process in more general terms: “In order for 
a program to improve itself substantially it would have to have at least a rudimentary understanding 
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short, predictive algorithms may evolve to develop an artificial 
intelligence (AJ) that guides their evolution. 

The goals of AI are twofold. From an engineering perspective, AI is 
the “science of making machines do things that would require 
intelligence if done by” persons.”° By contrast, the cognitive perspective 
envisions AI as designing systems that work the way the human mind 
does.” The distinct goals of the accounts of AI matter. The engineering 
perspective aims to perform a certain task (e.g., to minimize defaults, as 
in the credit context), regardless of how it does so.” This is the classic 
“black box,” which converts inputs to outputs without revealing how it 
does so. Alternatively, the cognitive perspective aspires for AI to 
replicate human capacities, such as emotions and self-consciousness, 
though often it falls short.” If scoring systems are to fulfill engineering 
goals and retain human values of fairness, we need to create backstops 
for human review. 

Algorithmic scoring should not proceed without expert oversight. 
This debate is already developing in the field of “killer robots,” where 
military theorists have described the following distinctions in terms of 
potentially autonomous, AI-driven weapons: 


e Human-in-the-Loop Weapons: Robots that can select 
targets and deliver force only with a human command; 


of its own problem-solving process and some ability to recognize an improvement when it found 
one. There is no inherent reason why this should be impossible for a machine.” Marvin L. Minsky, 
Artificial Intelligence, SCI. AM., Sept. 1966, at 246, 260. 


26. SAMIR CHOPRA & LAURENCE F. WHITE, A LEGAL THEORY OF AUTONOMOUS ARTIFICIAL 
AGENTS 5 (2011) (internal quotation marks omitted). 


27. Id. Ryan Calo has been a thought leader in integrating different conceptions of AI to 
contemporary privacy problems and the field of robotics. See, e.g., M. Ryan Calo, Robots and 
Privacy, in ROBOT ETHICS: THE ETHICAL AND SOCIAL IMPLICATIONS OF ROBOTICS 187 (Patrick Lin 
et al. eds., 2012); M. Ryan Calo, Open Robotics, 70 MD. L. REV. 571 (2011); M. Ryan Calo, 
Peeping Hals, 175 ARTIFICIAL INTELLIGENCE 940 (2011). 


28. Anderson, supra note 24. 


29. CHOPRA & WHITE, supra note 26, at 5 (“There are two views of the goals of artificial 
intelligence. From an engineering perspective, as Marvin Minsky noted, it is the ‘science of making 
machines do things that would require intelligence if done by men.’ From a cognitive science 
perspective, it is to design and build systems that work the way the human mind does. In the former 
perspective, artificial intelligence is deemed successful along a performative dimension; in the 
latter, along a theoretical one. The latter embodies Giambattista Vico’s perspective of verum et 
factum convertuntur, ‘the true and the made are...convertible’; in such a view, artificial 
intelligence would be reckoned the laboratory that validates our best science of the human mind. 
This perspective sometimes shades into the claim artificial intelligence’s success lies in the 
replication of human capacities such as emotions, the sensations of taste, and self-consciousness. 
Here, artificial intelligence is conceived of as building artificial persons, not just designing systems 
that are ‘intelligent.’” (alteration in original) (citations omitted)). 
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e Human-on-the-Loop Weapons: Robots that can select 
targets and deliver force under the oversight of a human 
operator who can override the robots’ actions; and 

e Human-out-of-the-Loop Weapons: Robots that are 
capable of selecting targets and delivering force without 
any human input or interaction. 

Human rights advocates and computer scientists contend that 
“Human-out-of-the-Loop Weapons” systems violate international law 
because AI systems cannot adequately incorporate the rules of 
distinction (“which requires armed forces to distinguish between 
combatants and noncombatants”) and proportionality.*' They create a 
“responsibility gap” between commanders and killing machines.” Such 
decisions arguably are the unique responsibility of persons using 
holistic, non-algorithmic judgment to oversee complex and difficult 
situations.” 

Just as automated killing machines violate basic legal norms, 
stigmatizing scoring systems at the least should be viewed with caution. 
We should not simply accept their predictions without understanding 
how they came about, and assuring that some human reviewer can 
respond to serious concerns about their fairness or accuracy. 

Scoring systems are often assessed from an engineering perspective, 
as a calculative risk management technology making tough but 
ultimately technical rankings of populations as a whole. We call for the 
integration of the cognitive perspective of AI. In this Article, we explore 
the consequences to human values of fairness and justice when scoring 
machines make judgments about individuals. Although algorithmic 
predictions harm individuals’ life opportunities often in arbitrary and 
discriminatory ways, they remain secret.** Human oversight is needed to 





30. See Losing Humanity: The Case Against Killer Robots, HUM. RTS. WATCH (Int’] Human 
Rights Clinic, Harvard Law Sch., Cambridge, Mass.), Nov. 2012, at 2, available at 
http://www. hrw.org/sites/default/files/reports/arms1112_ForUpload.pdf. 

31. Id. at 30. 

32. Id. at 42. 


33. See JOSEPH WEIZENBAUM, COMPUTER POWER AND HUMAN REASON: FROM JUDGMENT TO 
CALCULATION 227 (1976) (insisting that we should not delegate to computers “tasks that demand 
wisdom”). This is not to overstate the analogy of a low credit score to the kind of liberty deprivation 
at stake in weaponry. The stakes of war are far greater than being sure that an individual can be 
charged a higher interest rate. Nonetheless, under the Mathews v. Eldridge, 424 U.S. 319 (1976), 
calculus familiar to all students of administrative and constitutional law, id. at 332-39, we should 
not reject the targeting analogy as more-and-more predictive algorithms impact more-and-more 
aspects of our lives. 


34. On the importance of transparency and accountability in algorithms of powerful internet 
intermediaries, see Bracha & Pasquale, supra note 12; Frank Pasquale, Beyond Innovation and 
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police these problems. 

This Article uses credit scoring as a case study to take a hard look at 
our scoring society more generally. Part II describes the development of 
credit scoring and explores its problems. Evidence suggests that what is 
supposed to be an objective aggregation and assessment of data—the 
credit score—is arbitrary and has a disparate impact on women and 
minorities. Critiques of credit scoring systems come back to the same 
problem: the secrecy of their workings and growing influence as a 
reputational metric. Scoring systems cannot be meaningfully checked 
because their technical building blocks are trade secrets. Part III argues 
that transparency of scoring systems is essential. It borrows from our due 
process tradition and calls for “technological due process” to introduce 
human values and oversight back into the picture. Scoring systems and 
the arbitrary and inaccurate outcomes they produce must be subject to 
expert review. 


I. CASE STUDY OF FINANCIAL RISK SCORING 


Credit scores can make or break the economic fate of millions of 
individuals. New York Times business reporter Joe Nocera observes that 
while a “credit score is derived after an information-gathering process 
that is anything but rigorous,”” it “[e]ssentially . . . has become the only 
thing that matters anymore to the banks and other institutions that 
underwrite mortgages.”*° In this Part, we will provide a brief 
background on credit scoring systems and explore their core problems. 


A. A (Very) Brief History of Credit Scoring Systems 


Credit scoring in the United States has developed over six decades.” 
Initially, retail and banking staff assessed borrowers’ trustworthiness.** 
In time, experts were entrusted to make lending decisions.” After World 





Competition: The Need for Qualified Transparency in Internet Intermediaries, 104 Nw. U. L. REV. 
105 (2010) [hereinafter Pasquale, Beyond Innovation and Competition]; Frank Pasquale, Taking on 
the Known Unknowns, CONCURRING OPINIONS (Aug. 12, 2007), 
http://www.concurringopinions.com/archives/2007/08/taking on_the_k.html. 

35° Joe Nocera, Credit Score is the Tyrant in Lending, N.Y. TIMES, July 24, 2010, at B1. 

36° Id. (reporting statement of Deb Killian, Board Member, National Association of Mortgage 
Brokers). 

37. Abdou & Pointon, supra note 10, at 59. 

38. See ROBERT D. MANNING, CREDIT CARD NATION: THE CONSEQUENCES OF AMERICA’S 
ADDICTION TO CREDIT 83 (2000). 

39. EVANS CLARK, FINANCING THE CONSUMER 1-15, 114-56, 358 (1930). 
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War II, specialized finance companies entered the mix.“ 

In 1956, the firm Fair, Isaac & Co. (now known as FICO) devised a 
three-digit credit score, promoting its services to banks and finance 
companies." FICO marketed its scores as predictors of whether 
consumers would default on their debts.” FICO scores range from 300 
to 850. FICO’s scoring system remains powerful, though credit bureaus 
(“consumer reporting agencies”)*? have developed their own scoring 
systems as well.“ 

Credit scores legitimated the complex securities at the heart of the 
recent financial crisis.” In the mid-2000s, the credit score was the key 
connecting ordinary U.S. homeowners with international capital 
investors eager to invest in highly rated securities.*° When investors 
purchased a mortgage-backed security, they bought the right to a stream 
of payments.“ The mortgagor (borrower) shifted from paying the 





40. Stan Sienkiewicz, Credit Cards and Payment Efficiency 3 (Aug. 2001) (unpublished 
discussion paper), available at http://www.philadelphiafed.org/consumer-credit-and- 
payments/payment-cards-center/publications/discussionpapers/2001/PaymentEfficiency_092001. 
pdf. 

41. Martha Poon, Scorecards as Market Devices for Consumer Credit: The Case of Fair, Isaac & 
Company Incorporated, 55 SOC. REV. MONOGRAPH 284, 288 (2007); Fair, Isaac and Company 
History, FUNDINGUNIVERSE, _http://www.fundinguniverse.com/company-histories/Fair-Isaac-and- 
Company-Company-History.html (last visited Feb. 8, 2014). 

42. On predicting “derogatory events,’ see The FICO Score, THECREDITSCORINGSITE, 
http://www.creditscoring.com/creditscore/fico/ (last visited Feb. 8, 2014). 


43. For a definition of credit bureau, see Elkins v. Ocwen Federal Savings Bank Experian 
Information Solutions, Inc., No. 06 CV 823, 2007 U.S. Dist. LEXIS 84556, at *36-37 (N.D. Ill. 
Nov. 13, 2007) (explaining that credit bureaus and consumer reporting agencies regularly receive 
updates on a consumer’s credit relationships from their data furnishers, such as banks, mortgage 
companies, debt collectors, credit card issuers, department stores and others, and produce reports 
that contain highly sensitive and personal details about a consumer’s finances, including account 
numbers, loan balances, credit limits, and payment history). 


44. A court case describes the fight between FICO and credit bureaus over the credit bureaus’ 
development of their own scoring systems. See Fair Isaac Corp. v. Experian Info. Solutions, Inc., 
645 F. Supp. 2d 734 (D. Minn. 2009), aff'd, 650 F.3d 1139 (8th Cir. 2011). In such cases, courts use 
protective orders to ensure the confidentiality of trade secrets. See, e.g., Textured Yarn Co. v. 
Burkart-Schier Chem. Co., 41 F.R.D. 158 (E.D. Tenn. 1966). 

45. Martha Poon, From New Deal Institutions to Capital Markets: Commercial Consumer Risk 
Scores and the Making of Subprime Mortgage Finance, 34 ACCT. ORGS. & SOC’Y, 654, 662 (2009). 
In 1995, government-sponsored entities Fannie Mae and Freddie Mac announced that borrowers 
needed a credit score of at least 660 (on FICO’s scale of 300 to 850) for loans to qualify for the 
status of “prime investment.” Jd. at 663. Those below 660 were relegated to “subprime” offerings. 
Id. at 664. 

46. Id. at 655. 


47. Chris Wilson, What Is a Mortgage-Backed Security?, SLATE (Mar. 17, 2008, 7:09 PM), 
http://www.slate.com/articles/news_and_politics/explainer/2008/03/what_is_a_mortgagebacked_se 
curity. html. 
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original mortgagee (lender) to paying the purchaser of the mortgage- 
backed security, usually through a servicer.“ Fannie Mae, Freddie Mac, 
and networks of investors helped promote the credit score as a 
“calculative risk management technolog[y].”” 

Pricing according to credit scores had a dark side. The credit score 
moved the mortgage industry from “control-by-screening,” which aimed 
to eliminate those who were unlikely to pay back their debts, to “control- 
by-risk characterized by a segmented accommodation of varying credit 
qualities.” Abuses piled up. Subprime-structured finance generated 
enormous fees for middlemen and those with “big short” positions, 
while delivering financial ruin to many end-purchasers of mortgage- 
backed securities and millions of homebuyers.” 


B. The Problems of Credit Scoring 


Long before the financial crisis, critics have questioned the fairness of 
credit scoring systems. According to experts, the scores’ “black box” 
assessments were “inevitably subjective and value-laden,” yet seemingly 
“incontestable by the apparent simplicity of [a] single figure.”*” There 
are three basic problems with credit scoring systems: their opacity, 
arbitrary results, and disparate impact on women and minorities. 


1I. Opacity 


Behind the three-digit score (whether a raw FICO score, or another 
commercial credit score) is a process that cannot be fully understood, 
challenged, or audited by the individuals scored or even by the 
regulators charged with protecting them. Credit bureaus routinely deny 
requests for details on their scoring systems.” No one outside the 
scoring entity can conduct an audit of the underlying predictive 
algorithms. Algorithms, and even the median and average scores, 


48. See Mortgage-Backed Securities, PIMCO (Feb. 2009), http://www.pimco.com/EN/Education/ 
Pages/MortgageBackedSecurities.aspx. 

49. Poon, supra note 45, at 654. 

50. Id. at 658 (emphasis omitted). 

51. See generally MICHAEL LEWIS, THE BIG SHORT: INSIDE THE DOOMSDAY MACHINE (2010). 

52. Donncha Marron, ‘Lending by Numbers’: Credit Scoring and the Constitution of Risk Within 


American Consumer Credit, 36 ECON. & Soc’y 103, 111 (2007). For another black-box analogy, 
see Poon, supra note 45, at 658. 

53. See Index of Letters, CREDITSCORING, http://www.creditscoring.com/letters/ (last visited Feb. 
9, 2014) (documenting a series of letter requests and stonewalling responses). There have been 
repeated efforts by the bureaus to resist mandatory disclosure, or even filing the models with states. 

54. See Fair Isaac Corp. v. Equifax, Inc., No. 06-4112, 2007 U.S. Dist. LEXIS 71187 (D. Minn. 
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remain secret. 

The lack of transparency of credit-scoring systems leaves consumers 
confounded by how and why their scores change.” FICO and the credit 
bureaus do not explain the extent to which individual behavior affects 
certain categories.” Consumers cannot determine optimal credit 
behavior or even what to do to avoid a hit on their scores. 

FICO and credit bureaus do, however, announce the relative weight of 
certain categories in their scoring systems.” For example, “credit 
utilization” (how much of a borrower’s current credit lines are being 
used) may be used. But the optimal credit utilization strategy is unclear. 
No one knows whether, for instance, using twenty-five percent of one’s 
credit limit is better or worse than using fifteen percent. An ambitious 
consumer could try to reverse-engineer credit scores, but such efforts 
would be expensive and unreliable.” 

As various rankings proliferate, so do uncertainties about one’s 
standing.” Even the most conscientious borrower may end up surprised 
by the consequences of his actions. Responding to the confusion, books, 
articles, and websites offer advice on scoring systems. Amazon offers 
dozens of self-help books on the topic, each capitalizing on credit 
scoring’s simultaneously mystifying and meritocratic reputation.” 
Hucksters abound in the cottage industry of do-it-yourself credit repair. 


2. Arbitrary Assessments 


Credit-scoring systems produce arbitrary results, as demonstrated by 


Sept. 25, 2007); see also Public Comment Letter from Greg Fisher, Creditscoring.com, to the Board 
of Governors of the Federal Reserve (Sept. 17, 2004), available at 
http://www.federalreserve.gov/SECRS/2004/October/20041014/OP-1209/OP-1209_106_1.pdf. 

55. Yuliya Demyanyk, Your Credit Score Is a Ranking, Not a Score, FED. RES. BANK 
CLEVELAND (Nov. 16, 2010), http://www.clevelandfed.org/research/commentary/2010/2010- 
16.cfm. 

56. Credit Checks & Inquiries, MYFICO, _ http://www.myfico.com/crediteducation/ 
creditinquiries.aspx (last visited Feb. 9, 2014). 

57. See, e.g., What’s in My FICO Score?, MYFICO, http://www.myfico.com/CreditEducation/ 
WhatsInY ourScore.aspx (last visited Feb. 9, 2014). 

58. See Dean Foust & Aaron Pressman, Credit Scores: Not-So-Magic Numbers, BLOOMBERG 
BUSINESSWEEK (Feb. 6, 2008), http://www.businessweek.com/stories/2008-02-06/credit-scores-not- 
so-magic-numbers. 

59. See, e.g., Sue Kirchhoff & Sandra Block, Alternative Credit Scores Could Open Door for 
Loans, USA TODAY (May 16, 2006, 10:01 PM), http://usatoday30.usatoday.com/money/ 
perfi/credit/2006-05-16-credit-scores-usat_x.htm. 

60. See, e.g., Owing! 5 Lessons on Surviving Your Debt Living in a Culture of Credit [Kindle 
Edition], | AMAZON.COM, __http://www.amazon.com/Lessons-Surviving-Living-Culture-Credit- 
ebook/dp/BOOC2BMN3W (last visited Feb. 12, 2014). 
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the wide dispersion of credit scores set by the commercial credit 
bureaus.°! In a study of 500,000 files, 29% of consumers had credit 
scores that differed by at least 50 points between the three credit 
bureaus.” Barring some undisclosed, divergent aims of the bureaus, 
these variations suggest a substantial proportion of arbitrary 
assessments. 

Evidencing their arbitrary nature, credit-scoring systems seemingly 
penalize cardholders for their responsible behavior. In 2010, a 
movement called “Show Me the Note” urged homeowners to demand 
that servicers prove they had legal rights to mortgage payments.™ Given 
the unprecedented level of foreclosure fraud, homeowners rightfully 
wanted to know who owned the stream of payments due from their 
mortgage. 

A sensible credit-scoring system would reward those who had taken 
the trouble to demand accurate information about their mortgage.® The 
opposite, however, has happened. In one reported case, a homeowner 
who followed all the instructions on the “Where’s the Note” website 
allegedly experienced a “40 point hit” on his credit score. In the 
Kafkaesque world of credit scoring, merely trying to figure out possible 
effects on one’s score can reduce it. 

Of course, any particular case can be dismissed as an outlier, an 
isolated complaint by an unfortunate person. But this example is the tip 
of the iceberg. Over the past twenty years, a critical mass of complaints 


61. Carolyn Carter et al., The Credit Card Market and Regulation: In Need of Repair, 10 N.C. 
BANKING INST. 23, 41 (2006). Even after bureaus adopted the advanced “VantageScore” system, 
“70% of the dispersion remains.” Peter Coy, Giving Credit Where Credit is Due, BLOOMBERG 
BUSINESSWEEK (Mar. 14, 2006), http://www.businessweek.com/stories/2006-03-14/giving-credit- 
where-credit-is-duebusinessweek-business-news-stock-market-and-financial-advice (“It has been 
highly frustrating to lenders—and to borrowers—that the same person could get drastically different 
credit scores from different bureaus.”). 

62. Carter et al., supra note 61, at 41. 

63. See, e.g., The Secret Score Behind Your Auto Insurance, CONSUMER REP., Aug. 2006, at 43 
(noting that “insurance scores can penalize consumers who use credit reasonably”). 

64. See, e.g., Mathew Hector, Standing, Securitization, and “Show Me the Note,” SULAIMAN 
LAW GRP., http://www.sulaimanlaw.com/Publications/Standing-Securitization-and-Show-Me-The- 
Note.shtml (last visited Feb. 9, 2014). 

65. For background on foreclosure fraud, see generally YVES SMITH, WHISTLEBLOWERS REVEAL 
How BANK OF AMERICA DEFRAUDED HOMEOWNERS AND PAID FOR A COVER UP—ALL WITH THE 
HELP OF “REGULATORS” (2013). 

66. Cf Deltafreq, Comment to Where’s the Note? Leads BAC to Ding Credit Score, THE BIG 
PICTURE (Dec. 14, 2010, 11:03 AM), http://www.ritholtz.com/blog/2010/12/note-bac-credit-score/. 

67. Barry Ritholtz, Where’s the Note? Leads BAC to Ding Credit Score, THE BIG PICTURE (Dec. 
14, 2010, 9:15 AM), http://www.ritholtz.com/blog/2010/12/note-bac-credit-score/. 
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about credit scoring has emerged.“ Cassandra Jones Havard contends 
that scoring models may play an integral role in discriminatory lending 
practices.” Another commentator has charged that they enabled reckless 
securitizations that had devastating systemic impact.” 

In many accounts of the financial crisis, credit scores exerted a baleful 
influence, rationalizing lending practices with ersatz quantification. As 
Amar Bhide argued, the idea of “one best way” to rank credit applicants 
flattened the distributed, varying judgment of local loan officers into the 
nationwide credit score—a number focused on persons rather than 
communities."' Like monocultural-farming technology vulnerable to one 
unanticipated bug, the converging methods of credit assessment failed 
spectacularly when macroeconomic conditions changed. The illusion of 
commensurability and solid valuation provided by the models that 
mortgage-based securities were based on helped spark a rush for what 
appeared to be easy returns, exacerbating both boom and bust dynamics. 


3. Disparate Impact 


Far from eliminating existing discriminatory practices, credit-scoring 
algorithms instead grant them an imprimatur, systematizing them in 
hidden ways.” Credit scores are only as free from bias as the software 


68. See, e.g., Kevin Simpson, Insurers’ Use of Credit Reports Rankles Many, DENVER POST, 
Aug. 20, 2003, at Al (“Credit-scoring has been one of the components responsible for an ‘alarming 
trend’ of increased complaints to regulators over the past three years . . . .”). 

69. Cassandra Jones Havard, “On The Take”: The Black Box of Credit Scoring and Mortgage 
Discrimination, 20 B.U. PUB. INT. L.J. 241, 247 (2011) (arguing that credit scoring if unchecked is 
an intrinsic, established form of discrimination very similar to redlining). 


70. Brenda Reddix-Smalls, Credit Scoring and Trade Secrecy: An Algorithmic Quagmire or How 
the Lack of Transparency in Complex Financial Models Scuttled the Finance Market, 12 U.C. 
Davis Bus. L.J. 87 (2011). 


71. See generally AMAR BHIDE, A CALL FOR JUDGMENT: SENSIBLE FINANCE FOR A DYNAMIC 
ECONOMY (2010); Meredith Schramm-Strosser, The “Not So” Fair Credit Reporting Act: Federal 
Preemption, Injunctive Relief, and the Need to Return Remedies for Common Law Defamation to 
the States, 14 DUQ. Bus. L.J. 165, 169 (2012) (“A consumer’s reputation and credibility is 
determined not by personal interactions with others in a small community, but by examining credit 
files in an impersonal global world.”). 


72. Havard, supra note 69, at 247 (arguing that “credit scoring if unchecked is an intrinsic, 
established form of discrimination very similar to redlining”). Cf Citron & Pasquale, supra note 16, 
at 1459 (exploring how bias against groups can be embedded in fusion centers’ data-mining 
algorithms and spread through the information sharing environment). The EEOC, in a lawsuit filed 
against Kaplan, claimed that use of credit history would have a disparate, negative impact against 
minority job applicants because of the lower average credit score of these groups. Press Release, 
Equal Emp’t Opportunity Comm’n, EEOC Files Nationwide Hiring Discrimination Lawsuit Against 
Kaplan Higher Education Corp. (Dec. 21, 2010), available at http://www.eeoc.gov/eeoc/newsroom/ 
release/12-21-10a.cfm. 
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and data behind them.” Software engineers construct the datasets mined 
by scoring systems; they define the parameters of data-mining analyses; 
they create the clusters, links, and decision trees applied;” they generate 
the predictive models applied.” The biases and values of system 
developers and software programmers are embedded into each and every 
step of development.” 

Beyond the biases embedded into code, some automated correlations 
and inferences may appear objective but may reflect bias. Algorithms 
may place a low score on occupations like migratory work or low- 
paying service jobs. This correlation may have no discriminatory intent, 
but if a majority of those workers are racial minorities, such variables 
can unfairly impact consumers’ loan application outcomes. ”’ 

To know for sure, we would need access to the source code, 
programmers’ notes, and algorithms at the heart of credit-scoring 
systems to test for human bias, which of course we do not have.” Credit 
bureaus may be laundering discrimination into black-boxed scores, 
which are immune from scrutiny.” 

We are not completely in the dark though about credit scores’ impact. 
Evidence suggests that credit scoring does indeed have a negative, 
disparate impact on traditionally disadvantaged groups.® Concerns 
about disparate impact have led many states to regulate the use of credit 





73. See SHAWN FREMSTAD & AMY TRAUB, DEMOS, DISCREDITING AMERICA: URGENT NEED TO 
REFORM THE NATION’S CREDIT REPORTING INDUSTRY 11 (2011), available at 
http://www.demos.org/sites/default/files/publications/Discrediting America_Demos.pdf 
(“[D]isparities in the credit reporting system mirror American society’s larger racial and economic 
inequalities. [A] large body of research indicates that Americans with low incomes, and especially 
African Americans and Latinos, are disproportionately likely to have low credit scores.”). 

74. Zarsky, supra note 22, at 1518. 

75. Id. at 1519. 


76. Citron, supra note 20, at 1271 (discussing how administrative decision-making systems can 
embed bias into programs that is then applied to countless cases). 


77. Kenneth G. Gunter, Computerized Credit Scoring’s Effect on the Lending Industry, 4 N.C. 
BANKING INST. 443, 445, 451-52 (2000). 


78. Reddix-Smalls, supra note 70, at 91 (“As property, complex finance risk models often receive 
intellectual property proprietary protection. These proprietary protections may take the form of 
patents, copyrights, trade secrets, and sometimes trademarks.”). 


79. Cf. Robert E. Goodin, Laundering Preferences, in FOUNDATIONS OF SOCIAL CHOICE THEORY 
75 (Jon Elster & Aanund Hylland eds., 1986). 

80. BIRNY BIRNBAUM, INSURERS’ USE OF CREDIT SCORING FOR HOMEOWNERS INSURANCE IN 
OHIO: A REPORT TO THE OHIO CIVIL RIGHTS COMMISSION 2 (2003) (“Based upon all the available 
information, it is our opinion that insurers’ use of insurance credit scoring for underwriting, rating, 
marketing and/or payment plan eligibility very likely has a disparate impact on poor and minority 
populations in Ohio.”). 
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scores in insurance underwriting.*' The National Fair Housing Alliance 
(NFHA) has criticized credit scores for disadvantaging women and 
minorities.*” 

Insurers’ use of credit scores has been challenged in court for their 
disparate impact on minorities. After years of litigation, Allstate agreed 
to a multi-million dollar settlement over “deficiencies in Allstate’s credit 
scoring procedure which plaintiffs say resulted in discriminatory action 
against approximately five million African-American and Hispanic 
customers.”*? As part of the settlement, Allstate allowed plaintiffs’ 
experts to critique and refine future scoring models. 

If illegal or unethical discrimination influences credit scoring, 
members of disadvantaged groups will have difficulty paying their 
bills.” Their late payments could be fed into credit scoring models as 
neutral, objective indicia of reliability and creditworthiness.*° The very 
benchmark against which discriminatory practices are measured may 
indeed be influenced by discriminatory practices. 

The paucity of enforcement activity makes it hard to assess the 
effectiveness of the Equal Credit Opportunity Act (ECOA), which 
prohibits discrimination in lending, and Regulation B, which applies 
ECOA to credit scoring systems.*’ Regulation B requires that the 
reasons for a denial of credit/lending has to be related to—and 





81. Credit-Based Insurance Scoring: Separating Facts from Fallacies, NAMIC POL’Y BRIEFING 
(Natl Ass’n of Mut. Ins. Cos., Indianapolis, Ind.), Feb. 2010, at 1, available at 
http://iky.org/documents/NAMIC_Policy_ Briefing on Insurance Scoring Feb_2010.pdf. 

82. The Future of Housing Finance: The Role of Private Mortgage Insurance: Hearing Before 
the Subcomm. on Capital Mkts., Ins. & Gov’t Sponsored Enters. of the H. Comm. on Fin. Servs., 
111th Cong. 16 (2010) (statement of Deborah Goldberg, Hurricane Relief Program Director, The 
National Fair Housing Alliance). The NFHA has expressed concern that “the use of credit scores 
tends to disadvantage people of color, women, and others whose scores are often lower than those 
of white borrowers.” Id. at 57. The NFHA has also expressed “growing concern about how useful 
credit scores are for predicting loan performance and whether the financial sector is placing too 
much reliance on credit scores rather than other risk factors such as loan terms.” Id. 

83. Dehoyos v. Allstate, 240 F.R.D. 269, 275 (W.D. Tex. 2007). The parties settled after the Fifth 
Circuit decided that federal civil rights law was not reverse preempted by the McCarran-Ferguson 
Act’s allocation of insurance regulatory authority to states. See Dehoyos v. Allstate Corp., 345 F.3d 
290, 299 (Sth Cir. 2003). The Equal Credit Opportunity Act (ECOA), which regulates lending 
practices, does not preempt state laws that are stricter than ECOA. 

84. Dehoyos, 240 F.R.D. at 276. 

85. See, e.g., Gunter, supra note 77, at 451-52. 


86. See generally BERNARD E. HARCOURT, AGAINST PREDICTION: PROFILING, POLICING, AND 
PUNISHING IN AN ACTUARIAL AGE (2007). 





87. Regulation B sets forth specific data that cannot be used in a credit scoring system, such as: 
public assistance status, likelihood that any person will bear or rear children, telephone listing, 
income because of a prohibited basis, inaccurate credit histories, and different standards for married 
and unmarried persons, race, color, religion, national origin, and sex. 12 C.F.R. § 202.5 (2013). 
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accurately describe—the factors actually scored by the creditor.** Based 
on the evidence we could uncover, cases are rare.®’ This is surely 
because litigation costs usually exceed the discounted present value of 
the monetary stakes involved. Fines and penalties probably are not large 
enough to deter troubling practices.” 


C. The Failure of the Current Regulatory Model 


Contemporary problems echo concerns about unreliable credit 
histories that prompted lawmakers to regulate the credit industry.” In 
1970, Congress passed the Fair Credit Reporting Act (FCRA)” “because 
it was worried that growing databases [of personal information] could be 
used in ways that were invisible and harmful to consumers.”” As 
Priscilla Regan notes, the FCRA was the first information privacy 
legislation in the United States.” 

The FCRA obligates credit bureaus and all other “consumer reporting 
agencies” to ensure that credit histories are accurate and relevant.” 
Consumers have the right to inspect their credit records, to demand 
corrections, and to annotate their records if disputes cannot be 
resolved. From lawmakers, however, industry extracted a major 


88. 12 C.F.R. § 202.9(b)(2). Furthermore, no factor that was a principal reason for adverse action 
may be excluded from the disclosure. Id. 

89. See Scott Ilgenfritz, Commentary, The Failure of Private Actions as an ECOA Enforcement 
Tool: A Call for Active Governmental Enforcement and Statutory Reform, 36 U. FLA. L. REV. 447, 
449 (1984) (“Despite congressional intent and the liberal relief provisions of the ECOA, there has 
been a relative dearth of private actions brought under the Act.”). 
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91. See ROBERT ELLIS SMITH, BEN FRANKLIN’S WEB SITE: PRIVACY AND CURIOSITY FROM 
PLYMOUTH ROCK TO THE INTERNET 319-20 (2004). 


92. 15 U.S.C. § 1681a-x (2012). 


93. Edith Ramirez, Chairwoman, Fed. Trade Comm’n, Keynote Address at the Technology 
Policy Institute Aspen Forum: Privacy Challenges in the Era of Big Data: A View from the 
Lifeguard’s Chair 3 (Aug. 19, 2013), available at 
http://www.ftc.gov/sites/default/files/documents/public_statements/privacy-challenges-big-data- 
view-lifeguard’ s-chair/1308 1 9bigdataaspen.pdf (transcript as prepared for delivery). 

94. PRISCILLA M. REGAN, LEGISLATING PRIVACY: TECHNOLOGY, SOCIAL VALUES, AND PUBLIC 
Pouicy 101 (1995). 


95. 15 U.S.C. § 1681e(b) (“Whenever a consumer reporting agency prepares a consumer report it 
shall follow reasonable procedures to assure maximum possible accuracy of the information 
concerning the individual about whom the report relates.”); see also id. § 1681a(f) (defining 
consumer reporting agency). See generally Reddix-Smalls, supra note 70, at 108-09 (discussing the 
history, purpose, and substance of the FCRA); The Fair Credit Reporting Act (FCRA) and the 
Privacy of Your Credit Report, ELECTRONIC PRIVACY INFO. CTR., http://epic.org/privacy/fcra/ (last 
visited Feb. 22, 2014) (same). 


96. See 15 U.S.C. § 16811 (“Procedure in Case of Disputed Accuracy”). 
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concession: immunity from defamation law.” By limiting the possible 
penalties for reputational injuries, the FCRA opened the door to tactics 
of stalling, obstinacy, and obfuscation by the credit industry.” 

What about credit scores? In 2003, the Fair and Accurate Credit 
Transactions Act (FACTA) required credit bureaus to disclose credit 
scores to individuals in exchange for a fee capped by the FTC. But the 
FACTA does not “require a consumer reporting agency to disclose to a 
consumer any information concerning credit scores or any other risk 
scores or predictors relating to the consumer,”” except for four “key 
factors” involved in credit decisions.’ Regrettably, those four factors 
do little to explain credit scores. Phrases like “type of bank accounts” 
and “type of credit references” are etiolated symbols, more suited for 
machine-to-machine interaction than personal explanation. Factors such 
as “too many revolving accounts” and “late payment” are commonplace 
even for those with high credit scores. ®! The law does not require credit 
scorers to tell individuals how much any given factor mattered to a 
particular score.” Looking forward, a consumer has no idea, for 
example, whether paying off a debt that is sixty days past due will raise 
her score. The industry remains highly opaque, with scored individuals 
unable to determine the exact consequences of their decisions. 

Although FCRA offers individuals a chance to dispute items on their 
credit history, it does not require credit bureaus to reveal the way they 
convert a history into a score." That is a trade secret; a designation 
offering powerful legal protections to companies that want to keep their 
business practices a secret." Despite such secrecy, we can draw some 
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conclusions about the black box society that credit scoring is creating. 
We have seen evidence that credit scores produce arbitrary results that 
may in fact further entrench inequality. 

Now, we turn to our proposals that aspire to bring procedural 
regularity and regulatory oversight to our scored society, while 
balancing the protection of other values, including the intellectual 
property of the developers of scoring technology. ‘°° 


Il. PROCEDURAL SAFEGUARDS FOR AUTOMATED SCORING 
SYSTEMS 


Predictive scoring may be an established feature of the Information 
Age, but it should not continue without check. Meaningful 
accountability is essential for predictive systems that sort people into 
“wheat” and “chaff,’ “employable” and “unemployable,” “poor 
candidates” and “hire away,” and “prime” and “subprime” borrowers. 

Procedural regularity is essential given the importance of predictive 
algorithms to people’s life opportunities—to borrow money, work, 
travel, obtain housing, get into college, and far more. Scores can become 
self-fulfilling prophecies, creating the financial distress they claim 
merely to indicate.'°° The act of designating someone as a likely credit 
risk (or bad hire, or reckless driver) raises the cost of future financing (or 
work, or insurance rates), increasing the likelihood of eventual 
insolvency or un-employability.'°’ When scoring systems have the 
potential to take a life of their own, contributing to or creating the 
situation they claim merely to predict, it becomes a normative matter, 
requiring moral justification and rationale. "°S 


used to calculate various kinds of credit scores are well-guarded trade secrets.”). 


105. For an in-depth exploration of the different ways private and public decisions have been 
hidden to our detriment, see generally FRANK PASQUALE, THE BLACK Box SOCIETY (forthcoming 
2014). 

106. See Michael Aleo & Pablo Svirsky, Foreclosure Fallout: The Banking Industry’s Attack on 
Disparate Impact Race Discrimination Claims Under the Fair Housing Act and the Equal Credit 
Opportunity Act, 18 B.U. PUB. INT. L.J. 1, 5 (2008) (“Ironically, because these borrowers are more 
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107. See id. 
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traditional notion that economics attempts to create freestanding representations of market processes 


2014] THE SCORED SOCIETY 19 


Scoring systems should be subject to fairness requirements that reflect 
their centrality in people’s lives. Private scoring systems should be as 
understandable to regulators as to firms’ engineers. However well an 
“invisible hand” coordinates economic activity generally speaking, 
markets depend on reliable information about the practices of firms that 
finance, rank, and rate consumers. Brandishing quasi-governmental 
authority to determine which individuals are worthy of financial 
backing, private scoring systems need to be held to a higher standard 
than the average firm. 

One of the great accomplishments of the legal order was holding the 
sovereign accountable for decisionmaking and giving subjects basic 
rights, in breakthroughs stretching from Runnymede to the Glorious 
Revolution of 1688 to the American Revolution. New algorithmic 
decisionmakers are sovereign over important aspects of individual lives. 
If law and due process are absent from this field, we are essentially 
paving the way to a new feudal order of unaccountable reputational 
intermediaries. '” 

How should we accomplish accountability? Protections could draw 
insights from what one of us has called “technological due process”— 
procedures ensuring that predictive algorithms live up to some standard 
of review and revision to ensure their fairness and accuracy.''” 
Procedural protections should apply not only to the scoring algorithms 
themselves (a kind of technology-driven rulemaking), but also to 
individual decisions based on algorithmic predictions (technology-driven 
adjudication). 

This is not to suggest that full due process guarantees are required as 
a matter of current law. Given the etiolated state of “state action” 





(which economic sociologists must then insist leaves out power, or cultural context, or the fullness 
of human agency).”). Some commentators have argued that we need to “recognize economics not as 
a (misguided) science of capitalism but as its technology, that is, as one of the active ingredients in 
the production and reproduction of the market order.” Marion Fourcade, Theories of Markets and 
Theories of Society, 50 AM. BEHAV. SCI. 1015, 1025 (2007). 


109. Our proposal for basic rights of citizens vis-4-vis scoring systems also finds support in the 
work of other scholars concerned about the extraordinary power of private companies. See, e.g., 
LorI ANDREWS, I KNOW WHO YOU ARE AND I SAW WHAT YOU DID: SOCIAL NETWORKS AND THE 
DEATH OF PRIVACY 189-91 (2012) (concluding with a proposal for a “Social Network 
Constitution”); REBECCA MACKINNON, CONSENT OF THE NETWORKED 240-41 (2012) (proposing 
ten principles of network governance); Jeffrey Rosen, Madison's Privacy Blind Spot, N.Y. TIMES, 
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doctrine in the United States, FICO and credit bureaus are not state 
actors; however, much of their business’s viability depends on the 
complex web of state supports and rules surrounding housing finance. 
Nonetheless, the underlying values of due process—transparency, 
accuracy, accountability, participation, and fairness'''—should animate 
the oversight of scoring systems given their profound impact on people’s 
lives. Scholars have built on the “technological due process” model to 
address private and public decision-making about individuals based on 
the mining of Big Data.''” 

We offer a number of strategies in this regard. Federal regulators, 
notably the Federal Trade Commission (FTC), should be given full 
access to credit-scoring systems so that they can be reviewed to protect 
against unfairness. Our other proposals pertain to individual decision- 
making based on algorithmic scores. Although our recommendations 
focus on credit scoring systems, they can extend more broadly to other 
predictive algorithms that have an unfair impact on consumers. 


A. Regulatory Oversight over Scoring Systems 


The first step toward reform will be to clearly distinguish between 
steps in the scoring process, giving scored individuals different rights at 
different steps. These steps include: 

1) Gathering data about scored individuals; 

2) Calculating the gathered data into scores; 

3) Disseminating the scores to decisionmakers, such as employers; 

4) Employers’ and others’ use of the scores in decisionmaking. 

We believe that the first step, data gathering, should be subject to the 
same strictures as FCRA—whatever the use of the data—once a firm has 
gathered data on more than 2,000 individuals. "° Individuals should have 
the right to inspect, correct, and dispute inaccurate data, and to know the 
sources (furnishers) of the data. Ironically, some data brokers now refuse 
to give out their data sources because of “confidentiality agreements” 
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with sources.''* That position (hiding behind privacy interests to violate 
consumer privacy) would not stand for consumer reporting agencies 
covered by FCRA. It should not stand for data brokers and the like. 

Second, at the calculation of data stage, ideally such calculations 
would be public, and all processes (whether driven by AI or other 
computing) would be inspectable. In some cases, the trade secrets may 
merit protection, and only a dedicated, closed review should be 
available. But in general, we need to switch the default in situations like 
this away from an assumption of secrecy, and toward the expectation 
that people deserve to know how they are rated and ranked. 

The third stage is more difficult, as it begins to implicate First 
Amendment issues. Given the Supreme Court’s ruling in Sorrell v. IMS 
Health Inc.''° and other rulings in cases involving the regulation of 
ranking systems, ''° courts may look askance at rules that limit the 
dissemination of data or scores.''’ Nevertheless, scored individuals 
should be notified when scores or data are communicated to an entity. 
That notification only increases speech; it does not restrict or censor 
communication. Coerced speech can implicate the First Amendment, but 
like Professor Neil Richards, we do not understand Sorrell to lay down a 
blanket rule that all data is speech.''® Transparency requirements are 
consistent with First Amendment doctrine. 

The fourth and final stage is the most controversial. We believe that— 
given the sensitivity of scoring and their disparate impact on vulnerable 
populations—scoring systems should be subject to licensing and audit 
requirements when they enter critical settings like employment, 
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insurance, and health care. Such licensing could be completed by private 
entities that are themselves licensed by the EEOC, OSHA, or the 
Department of Labor.''? This “licensing at one remove” has proven 
useful in the context of health information technology. ee 

Given scoring’s sensitivity, fair, accurate, and replicable use of data is 
critical. We cannot rely on companies themselves to “self-regulate” 
toward this end—they are obligated merely to find the most efficient 
mode of processing, and not to vindicate other social values including 
fairness. Licensing can serve as a way of assuring that public values 
inform this technology. 

Licensing entities could ensure that particularly sensitive data does 
not make it into scoring. For example, data brokers sell the names of 
parents whose child was killed in car crash,'”! of rape victims,” and of 
AIDS patients.'”’ Licensors could assure that being on such a list does 
not influence scoring. Public hearings could be held on other, troubling 
categories to gather input on whether they should be used for 
decisionmaking. Data brokers pigeonhole individuals on the basis of 
who-knows-what data and inferences. Before letting such monikers 
become de facto scarlet letters, "^ we need to have a broader societal 
conversation on the power wielded by data brokers and, particularly, the 
level of validity of such classifications. 

Many of our proposals would require legislation. We are under no 
illusions that Congress is presently inclined to promote them. However, 
as in the case of the massive health IT legislation of 2009 (HITECH), it 
is important to keep proposals “ready to hand” for those brief moments 
of opportunity when change can occur.” 
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Fortunately, the Federal Trade Commission does have statutory 
authority to move forward on several parts of the “scored society” 
agenda. The FTC can oversee credit-scoring systems under its authority 
to combat “unfair” trade practices under Section 5 of the Federal Trade 
Commission Act.” It can use this authority to develop much more 
robust oversight over credit scoring, which could then be a model for 
legislation for other scoring entities (or for state consumer protection 
authorities and state attorneys general with authority to promote fair 
information practices). 

“Unfair” commercial practices involve conduct that substantially 
harms consumers, or threatens to substantially harm consumers, which 
consumers cannot reasonably avoid, and where the harm outweighs the 
benefits.'’ In 2008, the FTC invoked its unfairness authority against a 
credit provider for basing credit reductions on an undisclosed behavioral 
scoring model that penalized consumers for using their credit cards for 
certain transactions, such as personal counseling.’ 

The FTC’s concerns about predictive algorithms have escalated with 
their increasing use. In March 2014, the FTC is hosting a panel of 
experts to discuss the private sector’s use of algorithmic scores to make 
decisions about individuals, including individuals’ credit risk with 
certain transactions, likelihood to take medication, and influence over 
others based on networked activities.” The FTC has identified the 
following topics for discussion: 

e How are companies utilizing these predictive scores? 
e How accurate are these scores and the underlying data 
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used to create them? 

e How can consumers benefit from the availability and 
use of these scores? 

e What are the privacy concerns surrounding the use of 
predictive scoring? 

e What consumer protections should be provided; for 
example, should consumers have access to these scores 
and the underlying data used to create them? 

e Should some of these scores be considered eligibility 
determinations that should be scrutinized under the Fair 
Credit Reporting Act?!°° 

FTC Chairwoman Edith Ramirez has voiced her concerns about 
algorithms that judge individuals “not because of what they’ve done, or 
what they will do in the future, but because inferences or correlations 
drawn by algorithms suggest they may behave in ways that make them 
poor credit or insurance risks, unsuitable candidates for employment or 
admission to schools or other institutions, or unlikely to carry out certain 
functions.”'*' In her view, predictive correlations amount to 
“arbitrariness-by-algorithm” for mischaracterized consumers. '*” 

Indeed, as Chairwoman Ramirez powerfully argues, decisions-by- 
algorithm require “transparency, meaningful oversight and procedures to 
remediate decisions that adversely affect individuals who have been 
wrongly categorized by correlation.” ° Companies must “ensure that by 
using big data algorithms they are not accidently classifying people 
based on categories that society has decided—by law or ethics—not to 
use, such as race, ethnic background, gender, and sexual orientation.” 4 

With Chairwoman Ramirez’s goals in mind and the FTC’s unfairness 
authority, the FTC should move forward in challenging credit-scoring 
systems. The next step is figuring out the practicalities of such 
enforcement. How can the FTC translate these aspirations into reality 
given that scoring systems are black boxes even to regulators? 


1. Transparency to Facilitate Testing 


The FTC should be given access to credit-scoring systems and other 
scoring systems that unfairly harm consumers. Access could be more or 
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less episodic depending on the extent of unfairness exhibited by the 
scoring system. Biannual audits would make sense for most scoring 
systems; more frequent monitoring would be necessary for those which 
had engaged in troubling conduct. '*° 

We should be particularly focused on scoring systems which rank and 
rate individuals who can do little or nothing to protect themselves. The 
FTC’s expert technologists'® could test scoring systems for bias, 
arbitrariness, and unfair mischaracterizations. To do so, they would need 
to view not only the datasets mined by scoring systems'*’ but also the 
source code and programmers’ notes describing the variables, 
correlations, and inferences embedded in the scoring systems’ 
algorithms. '** 

For the review to be meaningful in an era of great technological 
change, the FTC’s technical experts must be able to meaningfully assess 
systems whose predictions change pursuant to AI logic. They should 
permitted to test systems to detect patterns and correlations tied to 
classifications that are already suspect under American law, such as race, 
nationality, sexual orientation, and gender. Scoring systems should be 
run through testing suites that run expected and unexpected hypothetical 
scenarios designed by policy experts." Testing reflects the norm of 
proper software development, and would help detect both programmers’ 
potential bias and bias emerging from the AI system’s evolution. “° 


2. Risk Assessment Reports and Recommendations 


Once the FTC evaluates credit-scoring systems to detect 
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“arbitrariness-by-algorithm”—as Chairwoman Ramirez astutely puts 
it—it should issue a Privacy and Civil Liberties Impact Assessment 
evaluating a scoring system’s negative, disparate impact on protected 
groups, arbitrary results, mischaracterizations, and privacy harms. '*! In 
those assessments, the FTC could identify appropriate risk mitigation 
measures. 

An important question is the extent to which the public should have 
access to the data sets and logic of predictive credit-scoring systems. We 
believe that each data subject should have access to all data pertaining to 
the data subject. Ideally, the logics of predictive scoring systems should 
be open to public inspection as well. There is little evidence that the 
inability to keep such systems secret would diminish innovation. The 
lenders who rely on such systems want to avoid default—that in itself is 
enough to incentivize the maintenance and improvement of such 
systems. There is also not adequate evidence to give credence to 
“gaming” concerns—i.e., the fear that once the system is public, 
individuals will find ways to game it. While gaming is a real concern in 
online contexts, where, for example, a search engine optimizer could 
concoct link farms to game Google or other ranking algorithms if the 
signals became public, the signals used in credit evaluation are far 
costlier to fabricate. '*” Moreover, the real basis of commercial success in 
“big data” driven industries is likely the quantity of relevant data 
collected in the aggregate—something not necessarily revealed or 
shared via person-by-person disclosure of data held and scoring 
algorithms used. 

We must also ensure that academics and other experts can comment 
on such scoring systems. Kenneth Bamberger and Deidre Mulligan 
argue that Privacy Impact Assessments required by the E-Government 
Act are unsuccessful in part due to the public’s inability to comment on 
the design of systems whose specifications and source codes remain 
obscured. ^ 
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As Tal Zarsky argues, the public could be informed about the datasets 
that predictive systems mine without generating significant social 
risks.'“* Zarsky demonstrates that—when it comes to “the collection of 
data and aggregation of datasets’—it is evident that “providing 
information regarding the kinds and forms of data and databases used in 
the analysis . . . generate[s] limited social risks . . . [usually only in the 
context of] secretive governmental datasets.” "45 

The more difficult question concerns whether scoring systems’ source 
code, algorithmic predictions, and modeling should be transparent to 
affected individuals and ultimately the public at large. Neil Richards and 
Jonathan King astutely explain that “there are legitimate arguments for 
some level of big data secrecy,” including concerns “connected to highly 
sensitive intellectual property and national security assets.”'*° But these 
concerns are more than outweighed by the threats to human dignity 
posed by pervasive, secret, and automated scoring systems. At the very 
least, individuals should have a meaningful form of notice and a chance 
to challenge predictive scores that harm their ability to obtain credit, 
jobs, housing, and other important opportunities. 


B. Protections for Individuals 


In constructing strategies for technological due process in scoring 
contexts, it is helpful to consider the sort of notice individuals are owed 
when governmental systems make adverse decisions about them. Under 
the Due Process Clause, notice must be “reasonably calculated” to 
inform individuals of the government’s claims against them.” The 
sufficiency of notice depends upon its ability to inform affected 
individuals about the issues to be decided, the evidence supporting the 
government’s position, and the agency’s decisional process.'** Clear 
notice decreases the likelihood that agency action will rest upon 
“incorrect or misleading factual premises or on the misapplication of 
tules.”' 
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Notice problems have plagued agency decision-making systems. 
Automated systems administering public benefits programs have 
terminated or reduced people’s benefits without any explanation.'*’ That 
is largely because system developers failed to include audit trails that 
record the facts and law supporting every decision made by the 
computer.'*! Technological due process insists that automated systems 
include immutable audit trails to ensure that individuals receive notice of 
the basis of decisions against them. '~” 


1. Notice Guaranteed by Audit Trails 


Aggrieved consumers could be guaranteed reasonable notice if 
scoring systems included audit trails recording the correlations and 
inferences made algorithmically in the prediction process. With audit 
trails, individuals would have the means to understand their scores. They 
could challenge mischaracterizations and erroneous inferences that led to 
their scores. 

Even if scorers successfully press to maintain the confidentiality of 
their proprietary code and algorithms vis-a-vis the public at large, it is 
still possible for independent third parties to review it. One possibility is 
that in any individual adjudication, the technical aspects of the system 
could be covered by a protected order requiring their confidentiality. 
Another possibility is to limit disclosure of the scoring system to trusted 
neutral experts. Those experts could be entrusted to assess the 
inferences and correlations contained in the audit trails. They could 
assess if scores are based on illegitimate characteristics such as race, 
nationality, or gender or on mischaracterizations. This possibility would 
both protect scorers’ intellectual property and individuals’ interests. 


2. Interactive Modeling 


Another approach would be to give consumers the chance to see what 
happens to their score with different hypothetical alterations of their 
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credit histories. Imagine an interface where each aspect of a person’s 
credit history is represented on a wiki." To make it more concrete, 
picture a consumer who is facing a dilemma. She sees on her credit 
report that she has a bill that is thirty days overdue. She could secure a 
payday loan to pay the bill, but she’d face a usurious interest rate if she 
takes that option. She can probably earn enough money working 
overtime to pay the bill herself in forty days. Software could give her an 
idea of the relative merits of either course. If her score dropped by 100 
points when a bill went unpaid for a total of sixty days, she would be 
much more likely to opt for the payday loan than if a mere five points 
were deducted for that term of delinquency. 

Just as the authors of the children’s series Choose Your Own 
Adventure helped pave the way to the cornucopia of interactive 
entertainment now offered today,” so, too, might creative customer 
relations demystify credit scoring. Interactive modeling, known as 
“feedback and control,” has been successfully deployed in other 
technical contexts by a “values in design” movement. !?® It has promoted 
automated systems that give individuals more of a sense of how future 
decisions will affect their evaluation. For example, Canada’s 
Immigration Bureau lets individuals enter various scenarios into a 
preliminary “test” for qualification as a permanent resident." The 
digital interface allows users to estimate how different decisions will 
affect their potential to become a Canadian citizen. Learning French or 
earning a graduate degree can be a great help to those in their thirties; on 
the other hand, some over sixty years old can do “everything right” and 
still end up with too few points to apply successfully. The public 
scorecard does not guarantee anyone admittance, and is revised over 
time. Nevertheless, it provides a rough outline of what matters to the 
scoring process, and how much. 
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Credit bureaus do need some flexibility to assess a rapidly changing 
financial environment. Any given score may be based on hundreds of 
shifting variables; a default may be much less stigmatizing in a year of 
mass foreclosures than in flush times. Credit bureaus may not be capable 
of predicting exactly how any given action will be scored in a week, a 
month, or a year. Nevertheless, they could easily “run the numbers” in 
old versions of the scoring software, letting applicants know how a given 
decision would have affected their scores on, for example, three different 
dates in the past. 

We need innovative ways to regulate the scoring systems used in the 
finance, insurance, and real estate industries, and perhaps might even 
consider a “public option” in credit scoring. Even if it were first only 
tried in an experimental set of loans, it could do a great deal of good. Ifa 
public system could do just as well as a private one, it would seriously 
deflate industry claims that scoring needs to be secretive—a topic 
explore in more depth in the next section. 


C. Objections 


Credit bureaus will object that transparency requirements—of any 
stripe—would undermine the whole reason for credit scores. Individuals 
could “game the system” if information about scoring algorithms were 
made public or leaked in violation of protective orders." Scored 
consumers would have ammunition to cheat, hiding risky behavior and 
routing around entities’ legitimate concerns such as fraud. 

We concede that incidental indicators of good credit can become 
much less powerful predictors if everyone learns about them. If it were 
to become widely known that, say, the optimal number of credit 
accounts is four, those desperate for a loan may be most likely to alter 
their financial status to conform with this norm. 

However, we should also ask ourselves, as a society, whether this 
method of judging and categorizing people—via a secretive, panoptic 





158. Odysseas Papadimitriou, Occupy Wall Street & Credit Score Reform, WALLETBLOG (Mar. 
21, 2012), http://www.walletblog.com/2012/03/credit-score-reform/ (“[T]he Occupiers are off-base 
in suggesting that we centralize credit scoring and make the underlying formulas public. This would 
only make it easier for people to game the system, which would make existing credit scores less 
useful to banks and lead more of them to create their own proprietary scores that consumers would 
have no way of accessing.”). But bureaus may have more “economic” incentives to keep their 
methods hidden. See Eric Pitter, The Law of Unintended Consequences: The Credit Scoring 
Implications of the Amended Bankruptcy Code—and How Bankruptcy Lawyers Can Help, 61 
CONSUMER FIN. L. Q. REP. 61, 65 (2007) (“CRAs have refused to disclose their credit scoring 
formula to anyone, even the Federal Reserve Board. The CRAs’ full exclusivity of their credit 
scoring model protects their niche and their unique role in the credit markets.”’). 


2014] THE SCORED SOCIETY 31 


sort—is appropriate. It has already contributed to one of the greatest 
financial crises in American history, legitimizing widespread subprime 
lending by purporting to scientifically rank individuals’ creditworthiness 
with extraordinary precision. Secretive credit scoring can needlessly 
complicate the social world, lend a patina of objectivity to dangerous 
investment practices, and encode discriminatory practices in 
impenetrable algorithms. "°? 

The benefits of secrecy are murkier than these costs. Moreover, the 
secrecy of credit scoring can impede incremental innovation: how can 
outsiders develop better scoring systems if they have no way of 
accessing current ones? Secret credit scoring can undermine the public 
good, since opaque methods of scoring make it difficult for those who 
feel—and quite possibly are—wronged to press their case. 

If scorers can produce evidence about the bad effects of publicity, that 
might justify keeping the correlations, inferences, and logic of scoring 
algorithms from the public at large. But that logic would not apply to the 
FTC or third-party experts who would be bound to keep proprietary 
information confidential. 

Another objection is that our proposal only works when the very 
existence of scoring systems is public knowledge, as in the case of credit 
scores. In non-credit contexts, entities are under no legal obligation to 
disclose scoring systems to the public generally and to impacted 
individuals specifically. Some scoring systems are not a secret because 
their business model is the sale of scores to private and public entities. 
Data brokers, for instance, rank, categorize, and score consumers on 
non-credit bases so they can avoid the obligations of FCRA.'® 

To be sure, it is impossible to challenge a scoring system that 
consumers do not even know exists. Secret scores about people’s health, 
employability, habits, and the like may amount to unfair practices even 
though they fall outside the requirements of FCRA. In that case, the FTC 
would have authority to require entities to disclose hidden scoring 
systems. 
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Of course, scoring systems that remain secret would be difficult for 
the FTC to identify and interrogate. Lawmakers could insist upon the 
transparency of scoring systems that impact important life opportunities. 
California, for instance, has been at the forefront of efforts to improve 
the transparency of businesses’ use of consumer information.'®’ The 
FTC has called upon federal lawmakers to pass legislation giving 
consumers access to the information that data brokers hold about 
them.'® In September 2013, Senate Commerce Committee Chairman 
Jay Rockefeller announced his committee’s investigation of the 
information collection and sharing practices of top data brokers. '® We 
are particularly supportive of such efforts—scoring systems can only be 
meaningfully assessed if they are known and subject to challenge. 


CONCLUSION 


Imagine a young woman who failed to get a job out of college, and 
that failure reduced her “employability” score used by potential 
employers to determine her fitness for work. She found part-time work 
at a fast food restaurant. Her credit score fell far below 600 without her 
even knowing it, perhaps because of inferences associated with certain 
low-paying jobs. Her low credit score caused further bad outcomes, 
cascading into ever more challenging life circumstances. Talent 
analytics companies categorized her as a “non-innovator” and “waste.” 
With low scores across countless measures, the young woman was 
unable to get a full-time job. 

To quote Wolff and De-Shalit, “without something like the type of 
action plan set out here, societies are destined to continue to reinforce 
patterns of entrenched privilege and disadvantage, widening gaps 
between rich and poor, and perpetuation of disadvantage.”!™ Michael 
Walzer’s social theory also provides a compelling argument against the 
“big data’s” promiscuous mashup of various data sources to deny 
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opportunities. °° Providing oversight over scoring systems that can cause 
negative spirals should be a critical aim of our legal system. Scoring 
systems have a powerful allure—their simplicity gives the illusion of 
precision and reliability. But predictive algorithms can be anything but 
accurate and fair. They can narrow people’s life opportunities in 
arbitrary and discriminatory ways. 

As a society, we have made commitments to protect consumers from 
serious harms that they have no means to prevent. We have also aspired 
to provide individuals with notice about important decisions made about 
them and a chance to challenge them. These commitments can help us 
develop a model of due process for scoring systems. Transparency is a 
crucial first step, first to the FTC who can interrogate scoring systems 
under their unfairness authority. Opening up the black box scoring 
systems to individuals or neutral experts representing them is key to 
permitting them to challenge “arbitrariness by algorithm.” Our 
recommendations are provisional, yet, we hope the FTC and interested 
lawmakers move forward in bringing procedural regularity and oversight 
into our scored society. 
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PREFACE 





Big data and associated technologies have enormous potential for positive impact in the United 
States, from augmenting the sophistication of online interactions to enhancing understanding 
of climate change to making advances in healthcare. These efforts, as well as the technological 
trends of always-on networked devices, ubiquitous data collection, cheap storage, sensors, and 
computing power, will sour broader use of big data. Our challenge is to support growth in the 
beneficial use of big data while ensuring that it does not create unintended discriminatory 
consequences. 


The Obama Administration’s Big Data Working Group released reports on May 1, 2014+ and 
February 5, 2015.” These reports surveyed the use of data in the public and private sectors and 
analyzed opportunities for technological innovation as well as privacy challenges. One 
important social justice concern the 2014 report highlighted was “the potential of encoding 
discrimination in automated decisions” —that is, that discrimination may “be the inadvertent 
outcome of the way big data technologies are structured and used.” Building on these prior 
reports and the 2014 study conducted by the President’s Council of Advisors on Science and 
Technology (PCAST), the Administration is further examining how big data is used in the public 
and private sectors.? Specifically, we are examining case studies involving credit, employment, 
education, and criminal justice to shed light on how using big data to expand opportunity has 
the potential to introduce bias inadvertently that could affect individuals or groups. As 
discussed in a report released by the Federal Trade Commission (FTC) earlier this year, big data 
provides opportunities for innovations that reduce discrimination and promote fairness and 
opportunity, including expanding access to credit in low-income communities, removing 
subconscious human bias from hiring decisions and classrooms, and providing extra resources 
to at-risk students. However, the FTC also emphasized the need to prevent such technologies 
from being used to deny low-income communities credit, perpetuate long-standing biases in 
employment, or exclude underserved communities from other benefits and opportunities.° 


This report examines several case studies from the spheres of credit and lending, hiring and 
employment, higher education, and criminal justice to provide snapshots of opportunities and 
dangers, as well as ways that government policies can work to harness the power of big data 
and avoid discriminatory outcomes. These are issues that strike at the heart of American 
values, which we must work to advance in the face of emerging, innovative technologies. 
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THE BIG DATA REVIEW AND RECENT PROGRESS 





Civil rights legislation of the last century responded to the reality that some Americans were 
being denied access to fundamental building blocks of opportunity and security, such as 
employment, housing, access to financial products, and admission to universities, on the basis 
of race, color, national origin, religion, sex, gender, sexual orientation, disability, or family 
status. Today, in the United States, anti-discrimination laws help enforce the tenet that all 
people are to be treated equally. These safeguards are important to protect all Americans 
against discrimination. Big data techniques have the potential to enhance our ability to detect 
and prevent discriminatory harm. But, if these technologies are not implemented with care, 
they can also perpetuate, exacerbate, or mask harmful discrimination. © 


In May 2014, a federal working group led by then-Counselor to the President John Podesta 
released a report on the emerging role of big data in our changing world and the opportunities 
and challenges that it presents.’ This was followed by an update on February 5, 2015.8 These 
reports highlight how big data tools are already being used to improve lives, make the economy 
work better, and save taxpayer dollars. In our increasingly networked world, the building blocks 
of big data are everywhere. We upload messages and photos over social media to stay 
connected to our friends; our phones transmit our specific locations to transportation apps; and 
information about who we are and what we are interested in is collected by a wide variety of 
retail, advertising, and analytics companies. Supplying data to these services enables a greater 
degree of improvement and customization, but this sharing also creates opportunities for 
additional uses of our data that may be unexpected, invasive, or discriminatory. As data-driven 
services become increasingly ubiquitous, and as we come to depend on them more and more, 
we must address concerns about intentional or implicit biases that may emerge from both the 
data and the algorithms used as well as the impact they may have on the user and society. 
Questions of transparency arise when companies, institutions, and organizations use 
algorithmic systems and automated processes to inform decisions that affect our lives, such as 
whether or not we qualify for credit or employment opportunities, or which financial, 
employment and housing advertisements we see. ° 


Ideally, data systems will contribute to removing inappropriate human bias where it has 
previously existed. We must pay ongoing and careful attention to ensure that the use of big 
data does not contribute to systematically disadvantaging certain groups. To avoid exacerbating 
biases by encoding them into technological systems, we need to develop a principle of “equal 
opportunity by design” —designing data systems that promote fairness and safeguard against 


discrimination from the first step of the engineering process and continuing throughout their 
lifespan. 


The 2014 report focused on the emerging role of big data in our changing world and the 
opportunities and challenges posed by such technology.” The report made a series of 
recommendations to the Federal government, among them to advance several pieces of key 
legislation that would further individual privacy in the digital economy. The Administration, 
Congress, and the public have made progress in acting on these recommendations; in 2015 we 
issued an Interim Progress Report illustrating the steps that have been taken thus far.*+ Further, 
the Council of Economic Advisers issued a report on price discrimination, exploring the 
processes that companies use to harvest information to in essence charge different consumers 
different prices for products and services rendered, and the Federal Trade Commission (FTC) 
included price discrimination in its analysis of issues relating to big data.‘ 


The purpose of this report is to advance our understanding of the problems that we can 
address with big data and algorithmic systems, along with the challenges that exist in deploying 
them. This new set of practices has great potential to increase access to opportunity and help 
overcome discrimination, if fairness in computational systems and ethical approaches to data 
analytics are the norm. At the same time, there are great risks that the very same innovations 
could perpetuate discrimination and unequal access to opportunity as the use of data expands. 
This report examines instances where big data methods and systems are being used in the 
public and private sectors in order to illustrate the potential for positive and negative outcomes 
and the extent to which “equal opportunity by design” safeguards may help address harms. 
These examples are meant to be a snapshot of the problems that data analytics can help to 
solve and the potential issues that its use might create, rather than an exhaustive look or set of 
recommendations on avoiding discrimination as big data becomes more central to the work of 
government and business. 


OPPORTUNITIES AND CHALLENGES IN BIG DATA 





The Assumption: Big Data is Objective 


It is often assumed that big data techniques are unbiased because of the scale of the data and 
because the techniques are implemented through algorithmic systems. However, it is a mistake 
to assume they are objective simply because they are data-driven.*3 


The challenges of promoting fairness and overcoming the discriminatory effects of data can be 
grouped into the following two categories: 


1) Challenges relating to data used as inputs to an algorithm; and 


2) Challenges related to the inner workings of the algorithm itself. 


Challenge 1: Inputs to an Algorithm 


The algorithmic systems of big data employ sophisticated processes. These processes need 
inputs. Consider how you might use a smart phone or GPS device to get the fastest route to a 
particular destination. To begin, the device needs at least three inputs: (1) where you are, (2) 
where you want to go, and (3) a map of the area. It then employs algorithms to calculate the 
fastest route from (1) to (2) and generate directions for how to proceed. At its most basic, this 
algorithmic system might simply draw upon map data for its inputs, such as the names, 
locations, and length of the roads in a given city. To compute the fastest route, the algorithmic 
system would then calculate which path on the map involved the fewest number of roads with 
the shortest road length. A more advanced routing system could include maximum speed limits 
for each road or even information about traffic congestion such as data on the speed of other 
drivers collected from those drivers’ mobile devices. For these more complex algorithmic 
systems, the inputs extend beyond a simple street map to potentially include many others, such 
as weather updates, historical traffic patterns, and the presence of disruptive events occurring 
nearby. 


The decision to use certain inputs and not others can result in discriminatory outputs. Some of 
the technical themes that can cause discriminatory outputs include: 


e Poorly selected data, where the designers of the algorithmic system decide that certain 
data are important to the decision but not others. In the “fastest route” example, the 
architect of the system might only include information about roads but not public 
transportation schedules or bike routes, thereby disadvantaging individuals who do not 
own a vehicle. Such issues can be regarded as qualitative errors, where human choices 
in the selection of certain datasets as algorithmic inputs over others are ill-advised. 
Careless choices of input might lead to biased results—in the “fastest route” example, 
results that might favor routes for cars, discourage use of public transport, and create 
transit deserts. Relatedly, designers might select data that is of too much or too little 
granularity, resulting in potentially discriminatory effects. 


e Incomplete, incorrect, or outdated data, where there may be a lack of technical rigor 
and comprehensiveness to data collection, or where inaccuracies or gaps may exist in 
the data collected. In the “fastest route” example, this could occur if, for instance, the 
algorithmic system does not update bus or train schedules regularly. Even if the system 
works perfectly in other respects, the resulting directions could again discourage use of 


public transportation and disadvantage those who have no viable alternatives, such as 
many lower-income commuters and residents. 


e Selection bias, where the set of data inputs to a model is not representative of a 
population and thereby results in conclusions that could favor certain groups over 
others. In the “fastest route” example, if speed data is collected only from the 
individuals that own smartphones, then the system’s results may be more accurate for 
wealthier populations with higher concentrations of smart phones and less accurate in 
poorer areas where smart-phone concentrations are lower.14 


e Unintentional perpetuation and promotion of historical biases, where a feedback loop 


causes bias in inputs or results of the past to replicate itself in the outputs of an 
algorithmic system. For instance, when companies emphasize “hiring for culture fit” in 
their employment practices, they may inadvertently perpetuate past hiring patterns if 
their current workplace culture is primarily based on a specific and narrow set of 
experiences. In a workplace populated primarily with young white men, for example, an 
algorithmic system designed primarily to hire for culture fit (without taking into account 
other hiring goals, such as diversity of experience and perspective) might 
disproportionally recommend hiring more white men because they score best on fitting 
in with the culture. 


Each of these issues is critical to take into account in designing systems to deliver services 
effectively, fairly, and ethically to consumers and community members, or to influence 
processes like credit-granting, hiring, housing allocation, and admissions.’° Transparency, 
accountability, and due process mechanisms are important components of ensuring that the 
inputs to an algorithmic system are accurate and appropriate. 


Challenge 2: The Design of Algorithmic Systems and Machine Learning 


For those who are not directly involved in the technical development of algorithms for large 
scale data systems, the end product of such as system can feel like a “black box” —an opaque 
machine that takes inputs, carries out some inscrutable process, and delivers unexplained 
outputs based on that process. The technical processes involved in algorithmic systems are 
typically unknown to a consumer, potential student, job candidate, defendant, or the public as 
they are often treated as confidential or proprietary to the entities that use them.?” Some 
systems even “passively” pre-screen potential candidates without notice as a preemptive effort 
to streamline decision-making processes at a later date.!® This lack of transparency means that 
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affected individuals—such as those who receive word that they will not receive a job offer, 
were denied admission to their college of choice, or will be denied a line of credit or lease— 
have limited ability to learn the reasons why such decisions were made and limited ability to 
detect and seek correction of any errors or bias if they do occur. It may even mean that certain 
individuals will be entirely excluded from certain opportunities—for instance, seeing particular 
advertisements for jobs, financial products, or educational opportunities and never discover 
that they were denied these opportunities.*? Such situations can be complex and difficult to 
address, especially if the outputs are relied upon again in subsequent determinations. At a 
minimum, it is important to encourage transparency, accountability, and due process 
mechanisms wherever possible in the use of big data. Without these safeguards, hard-to-detect 
flaws could proliferate. Such flaws include: 


e Poorly designed matching systems, which are intended to help find information, 
resources, or services. For example, search engines, social media platforms, and 
applications rely on matching systems to determine search results, what advertisements 
to display, and which businesses to recommend. These matching systems may result in 
discriminatory outcomes if the system designs are not kept current or do not take into 
account historical biases or blind spots within the data or the algorithms used.*+ 


e Personalization and recommendation services that narrow instead of expand user 


options, where detailed information about individual users might be collected and 
analyzed to infer their preferences, interests, and beliefs in order to point them to 
opportunities such as new music to download, videos to watch, price discounts, or 
products to purchase. Academic studies have shown that the algorithms used to 
recommend such content may inadvertently restrict the flow of information to certain 
groups, leaving them without the same opportunities for economic access and inclusion 
as others.” 


e Decision-making systems that assume correlation necessarily implies causation, whereby 
a programmer or the algorithmic system itself may assume that because two factors 


frequently occur together (e.g., having a certain income level and being of a particular 
ethnicity), there is necessarily a causal relationship between the two. Assuming a causal 
relationship in these circumstances can lead to discrimination. 


e Data sets that lack information or disproportionately represent certain populations, 


resulting in skewed algorithmic systems that effectively encode discrimination because 
of the flawed nature of the initial inputs. Data availability, access to technology, and 
participation in the digital ecosystem vary considerably, due to economic, linguistic, 
structural or socioeconomic barriers, among others. Unaddressed, this systemic flaw can 


reinforce existing patterns of discrimination by over-representing some populations and 
under representing others. 


An additional area that presents challenges for further study is a genre of computer science 
known as machine learning—the “science of getting computers to act without being explicitly 
programmed.”*? Complex and often inscrutable even at times to their programmers, machine 
learning models are starting to be used in areas such as credit offers, entrepreneurial funding, 
or hiring. As these methods continue to advance, it may become more difficult to explain or 
account for the decisions machines make through this process unless mechanisms are built into 
their designs to ensure accountability.24 Using the principle of “equal opportunity by design” 
and grounding engineering with sound ethical and professional best practices will also help 
mitigate discriminatory results over time and increase inclusion. 


Just as in other areas, programmers and data scientists may inadvertently or unconsciously 
design, train, or deploy big data systems with biases. Therefore, an important factor in 
implementing the “equal opportunity by design” principle is engaging with the field of “bias 
mitigation” to avoid building in the designers’ biases that are an inevitable product of their own 
culture and experiences in life.2? Research-based methods are emerging that can help reduce 
biases in decision-making around hiring, promotions, classroom grading, funding, social 
engagement, and more.”° Use of these methods can help stop biased big data systems from 
becoming the norm, instead of the exception. 


As improvements in the uses of big data and machine learning continue, it will remain 
important not to place too much reliance on these new systems without questioning and 
continuously testing the inputs and mechanics behind them and the results they produce. 
“Data fundamentalism” —the belief that numbers cannot lie and always represent objective 
truth—can present serious and obfuscated bias problems that negatively impact people’s 
lives.?7 


As we work to address challenges related to data inputs and the inner workings of algorithms, 
we must also pay attention to how to the products of these algorithmic systems are used, with 
an eye for ensuring that information about places, people, preferences, and more is used 
legally, ethically, and to advance democratic principles, such as equality and opportunity. 


CASE STUDIES IN THE USE OF BIG DATA 





In this section, we present four case studies in the use of big data analytics: (1) access to credit, 
(2) higher education, (3) employment, and (4) criminal justice. We describe the opportunities 
each case study presents for algorithmic systems to support personal, commercial, and 
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organizational missions, as well as the challenges that arise in utilizing the data without adverse 
impacts. 


Big Data and Access to Credit 


The Problem: Many Americans lack access to affordable credit due to thin or non- 
existent credit files. 


Access to fairly-priced and affordable credit is an important factor in enabling Americans to 
thrive economically, especially those working to enter the middle class. For decades, lenders 
have largely relied on credit scores, such as the FICO score, to decide whether and on what 
terms to make a loan. Credit scores represent a prediction of the likelihood that someone will 
have a negative financial event, such as defaulting on a loan, within a specific period of time. 
Traditionally, this prediction is made based on actual data about a particular person’s credit 
history, and turned into a score using algorithms developed from past lending experiences. 
While traditional credit scores serve many Americans well, according to a study by the 
Consumer Financial Protection Bureau (CFPB), as many as 11 percent of consumers are “credit 
invisible’”—they simply do not have enough up-to-date credit repayment history for the 
algorithm to produce a credit score.7° In addition, the CFPB found a strong relationship 
between income and a scorable credit record—30 percent of consumers in low-income 
neighborhoods are “credit invisible” and the credit records of another 15 percent are 
unscorable.? According to the CFPB, African-Americans and Latinos are more likely to be credit 
invisible, at rates of around 15 percent in comparison to 9 percent for whites.° The CFPB also 
found that an additional 13 percent of African-Americans and 12 percent of Latinos are 
unscorable, compared to 7 percent for whites.?? 


The Big Data Opportunity: Use of big data in lending can increase access to credit for the 
financially underserved. 


One possible approach to this problem is to use data analytics drawing on multiple sources of 
information to create more opportunity for consumers to gain access to better credit. As 
companies collect information and score individuals, especially those without sufficient or 
updated credit information, data may be useful in assessing credit risk. Some companies look at 
previously untapped data, such as phone bills, public records, previous addresses, educational 
background, and tax records, while others may consider less conventional sources, such as 
location data derived from use of cellphones, information gleaned from social media platforms, 
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purchasing preferences via online shopping histories, and even the speeds at which applicants 
scroll through personal finance websites. 


New tools designed with big data have potential to create alternative scoring mechanisms and 
new opportunities for access to credit for the tens of millions of Americans who do not have 
enough information in their credit files to receive a traditional credit score or who have an 
undeservedly low score. For example, many Americans regularly pay their phone and utility 
bills—payment records that predict creditworthiness. But phone and utility payment 
information is generally only reported as part of a credit history if a customer falls far behind, 
and therefore it only ever serves to penalize a consumer’s perceived creditworthiness. One 
study by the Policy and Economic Research Council looked at more than four million credit files 
and found that if both positive and negative utility and telecom payments were included, over 
70 percent of the unscorable files would become scorable and 64 percent of the “thin files” 
(files with very little other credit history) would see improved scores.*” The study also found 
that this change especially benefits low-income borrowers.?? 


The Big Data Challenge: Expanding access to affordable credit while preserving 
consumer rights that protect against discrimination in credit eligibility decisions. 


While big data has the ability to increase American’s access to affordable credit, if not used 
with care, it also has the potential to perpetuate, exacerbate, or mask discrimination. For 
example, consider a technology that would glean information from an individual’s social media 
connections and use social analytic systems to create an alternative credit score. While such a 
tool might expand access to credit for those underserved by the traditional market, it could also 
function to reinforce disparities that already exist among those whose social networks are, like 
them, largely disconnected from everyday lending.*4 Such tools could also raise questions about 
the ability of consumers to dispute an adverse decision or to correct inaccurate information. 
When such decisions are made within computationally-driven ‘black box' systems, traditional 
notions of transparency may fail to fully capture and disclose the information consumers need 
to understand the basis of such decisions and the role that various data played in determining 
their credit eligibility. 


The right to be informed about and to dispute the accuracy of the underlying data used to 
create a credit score is particularly important because credit bureaus have significant data 
accuracy issues, which are likely to be exacerbated by the use of new, fast-changing data 
sources. An FTC study found that 21 percent of its sample of consumers had a confirmed error 
on at least one of their three credit bureau reports.*° Expanding the data sources for credit 
scoring systems from long-collected items like collections notices and credit card payments to 
fast-changing, large-volume data like social media usage and GPS location information would 
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most likely increase the presence of factually inaccurate data, leading to scores that are not 
based on a consumers’ likelihood of delinquency.*° Additionally, as the number of data sources 
increases, the relationship to creditworthiness becomes more complex and dynamic, and 
therefore consumers may have more difficulty interpreting the notices required under FRCA 
and ECOA and identifying problems.?’ Consumers with less experience dealing with large 
institutions or complex data products may be particularly vulnerable to these data accuracy and 
transparency challenges. 


These concerns are not necessarily unique to the emerging data-analytics approaches to credit 
scoring. For example, in 2007 the FTC released a study of credit-based insurances scores finding 
that although there are substantial score disparities among ethnic groups, the scores are 
effective predictors of risk under automobile-insurance policies and are not simply proxies for 
race, ethnicity, sex, or other prohibited bases.?° As algorithms develop to measure 
creditworthiness in new ways, it will be critical to design and test them with similar concerns in 
mind and to guard against unintentionally using information that is a proxy for race, gender, or 
other protected characteristics .7° The limited research that does publicly exist has looked at 
whether the scores are effective predictors of delinquency, it has not examined whether new 
ways of evaluating credit worthiness adequately avoid considering proxies for traits like race or 
ethnicity.”° 


The shortage of studies on these new scoring products is a potential cause for concern because 
of the complexity and proprietary nature of these new products. If poorly implemented, 
algorithmic systems that utilize new scoring products to connect targeted marketing of credit 
opportunities with individual credit determinations could produce discriminatory harms. This is 
particularly concerning because the rapid pace of evolution in the credit sector, especially 
combined with ongoing advances in data science, makes it difficult for researchers and 
consumers alike to identify discrimination and take steps to prevent it.” 


Big Data and Employment 


The Problem: Traditional hiring practices may unnecessarily filter out applicants whose 
skills match the job opening. 


Beginning in the 1990s, a growing number of companies realized there was a new way to access 
and analyze a larger pool of applicants for a job opening rather than simply reviewing paper 
files.4 Resume-database websites provided a place where individuals and companies could gain 
access to opportunities and talent. To deal with the sudden influx of candidates, companies 
looking to hire also turned to new ways of rating applicants, using analytical tools to 
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automatically sort and identify the preferred candidates to move forward in a hiring process. 
With this change, the task of identifying and scoring applicants began to shift from industrial 
psychologists and recruiting specialists to computer scientists, through the use of algorithms 
and large data sets.*? 


Yet even as recruiting and hiring managers look to make greater use of algorithmic systems and 
automation, the inclination remains for individuals to hire someone similar to themselves, an 
unconscious phenomenon often referred to as “like me” bias, which can impede diversity.“ 
Algorithmic systems can be designed to help prevent this bias and increase diversity in the 
hiring process. Yet despite these goals, because they are built by humans and rely on imperfect 
data, these algorithmic systems may also be based on flawed judgments and assumptions that 
perpetuate bias as well. Because these technologies are new, rapidly changing, difficult to 
decipher, and often subject to proprietary protections, their determinations can be even more 
difficult to challenge. 


The Big Data Opportunity: Big data can be used to uncover or possibly reduce 
employment discrimination. 


Just as with credit scoring, data analytics can be beneficial to the workplace in helping match 
people with the right jobs. As discussed above, research has documented a “like me bias” or 
“affinity bias” in hiring; even well-intentioned hiring managers often choose candidates with 
whom they share characteristics. By contrast, algorithmically-driven processes have the 
potential to avoid individual biases and identify candidates who possess the skills that fit the 
particular job.*° 


Companies can use data-driven approaches to find potential employees who otherwise might 
have been overlooked based on traditional educational or workplace-experience requirements. 
Data-analytics systems allow companies to objectively consider experiences and skill sets that 
have a proven correlation with success. By looking at the skills that have made previous 
employees successful, a human-resources data system can “pattern match” in order to 
recognize the characteristics the next generation of hires should have.*” When fairness, ethics, 
and opportunity are a core part of the original design, large-scale data systems can help combat 
the implicit and explicit bias often seen in traditional hiring practices that can lead to 
problematic discrimination.*® Beyond hiring decisions, properly deployed, advanced algorithmic 
systems present the possibility of tackling age-old employment discrimination challenges, such 
as the wage gap or occupational segregation.” 
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The Big Data Challenge: Promoting fairness, ethics, and mechanisms for mitigating 
discrimination in employment opportunity. 


Data-analytics companies are creating new kinds of “candidate scores” by using diverse and 
novel sources of information on job candidates. These sources, and the algorithms used to 
develop them, sometimes use factors that could closely align with race or other protected 
characteristics, or may be unreliable in predicting success of an individual at a job. For example, 
workers who were unemployed for long periods during the recent economic downturn may 
have a harder time re-entering the workforce because candidate-scoring systems that consider 
“length of time since last job” can generate scores that send negative signals to potential 
employers that are unrelated to job performance. Similarly, one employment research firm 
found commuting distance to be one of the strongest predictors of how long a customer service 
employee will stay with a job.°° If algorithmic systems were trained to rely heavily on this factor 
without further consideration, they could end up discriminating against the candidates who, 
while otherwise qualified, happen to live in areas that are further away from the job than other 
candidates. While the factor of commuting distance was ultimately disregarded in this 
particular study out of concern for how highly it might correlate with race,°+ other employers 
might overlook such important factors. Other common hiring criteria, such as credit-worthiness 
(also the work of algorithms) and criminal records, compromise the validity of these tools if 
they inaccurately or inadequately reflect an individual’s qualifications. Implementation of such 
systems with an eye to their broader effects on fairness and equal opportunity is therefore 
essential. 


Finally, as described earlier, machine-learning algorithms can help determine what kinds of 
employees are likely to be successful by reviewing the past performance of existing employees 
or by analyzing the preferences of hiring managers as shown by their past decisions.°? But if 
those sources themselves contain historical biases, the scores may well replicate those same 
biases. For example, if machine-learning algorithms emphasize the age that a candidate 
became interested in computing compared to their peers, cultural messages and assumptions 
that associate computing with boys more often than with girls could promote environments 
where more boys than girls are exposed to computers at an earlier age, thereby skewing later 
hiring patterns toward more male hires, even though a company’s hiring goals may be focused 
on gender equality. ° Similar concerns could emerge regarding age discrimination, since older 
workers may be less likely to have grown up with home computers. Further, hiring algorithms 
that emphasize the need for a four-year college degree, or even a particular field of study or 
degree can leave out highly qualified, talented individuals who might not have those specific 
qualifications and could instead come into the job opportunity through on-the-job training or 
emerging alternative training and apprenticeship models—or who might have a four-year 
degree but in a different field than the ones sought by the algorithmic systems. These are the 
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types of factors that engineers and managers need to consider at the inception of designing 
analytical hiring systems and incorporate into the machine learning design work. “Equal 
opportunity by design” approaches are one way to promote these considerations. 


Companies have begun to filter their applicant pools for job openings using various human 
resources analytics platforms. It is critical to the fairness of American workplaces that all 
companies continue to promote fairness and ethical approaches to the use of data tools and 
ensure against the perpetuation of biases that could disfavor certain groups. Businesses also 
stand to benefit, because those that do not look beyond historical hiring patterns (even as 
mediated by an algorithm) will miss great candidates for important jobs. 


Big Data and Higher Education 


The Problem: Students often face challenges accessing higher education, finding 
information to help choose the right college, and staying enrolled. 


Prospective students and their families must grapple with assessing which of the many 
institutions of higher education available will best prepare them to achieve their goals. The 
decisions to pursue a degree, at which degree level, and at which institution all have a lasting 
impact on students and their futures. For example, obtaining a bachelor’s degree can increase 
total earnings by 84 percent over a lifetime relative to expected earnings for someone with a 
high school diploma. Similarly, differences in the price of attendance across institutions affect 
financial returns, and may lead to differences in the amount that students have to borrow, 
which may also affect their career decisions and personal lives in meaningful ways. Despite the 
importance of this decision, there is a surprising lack of clear, easy to use, and accessible 
information available to guide the students making these choices. 


At the same time, institutions of higher education collect and analyze tremendous amounts of 
data about their students and applicants. Before students arrive on campus, colleges use 
student information in recruitment, admissions, and financial aid decisions. After students 
enroll, some schools are using the data collected through the application process and in the 
classroom to tailor their students’ educational experiences.” The opportunities to use big data 
in higher education can either produce or prevent discrimination—the same technology that 
can help identify and serve students who are more likely to be in need of extra help can also be 
used to deny admissions or other opportunities based on the very same characteristics. 
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The Big Data Opportunity: Using big data can increase educational opportunities for the 
students who most need them. 


To address the lack of information about college quality and costs, the Administration has 
created a new College Scorecard to provide reliable information about college performance.°® 
The College Scorecard is a large step toward helping students and their families evaluate 
college choices. Never-before-released national data about post-college outcomes—including 
the most comparable and reliable data on the earnings of colleges’ alumni and new data on 
student debt—and student-loan repayment provides students, families, and their advisers with 
a more accurate picture of college cost and value. The data also encourages colleges to 
strengthen support that helps students persist in and complete college, and to provide 
increased opportunities for disadvantaged students to get a college education. Armed with this 
information, students and their families can make more informed decisions and better 
understand the opportunities and tradeoffs of their choices. 


In addition to data that the Department of Education has made available through the College 
Scorecard, institutions of higher education also present a unique environment to utilize 
innovations in big data for students once enrolled. Schools can use data they are already 
collecting to help track student progress. The opportunity for innovation lies in how schools use 
that existing data to create a tailored learning experience. Big data techniques are already 
helping students learn more effectively through tailored instruction, which can help overcome 
continuing disparities in learning outcomes, and providing extra help for those more likely to 
drop out or fail.°” 


Georgia State University is one example of a college using big data to drive student success. Its 
Graduation and Progression Success (GPS) Advising program, which started in 2013, is designed 
to keep the school’s more than 32,000 students on track for graduation. It tracks eight hundred 
different risk factors for each student on a daily basis. When a problem is detected, the 
university deploys proactive advising and timely interventions to provide the support that 
students need. At times the interventions are as simple—and essential—as ensuring the 
student has registered for the right courses; at other times, the system uses predictive analytics 
to make sure that the student's performance in a prerequisite course makes success likely at 
the next level. Since the GPS Advising initiative began in 2013, there have been nearly 100,000 
proactive interventions with Georgia State students based on the analytics-based alerts coming 
from the system. Over the past three years, Georgia State’s graduation rate has increased by 6 
percentage points, from 48 percent to 54 percent, when compared to the baseline year. The 
biggest gains have been enjoyed by at-risk populations. This year for the first time in Georgia 
State's history, first-generation, black, and Latino students as well as those on federally-funded 
Pell grants all graduated at rates at or above that of the student body overall, and Georgia State 


17 


now awards more Bachelor's degrees to black students than any non-profit college or university 
in the United States. Over the last two-years alone, Georgia State has reduced time-to-degree 
by an average of half a semester per student, saving students more than $10 million in tuition 
and fees.°° 


The Big Data Challenge: Administrators must be careful to address the possibility of 
discrimination in higher education admissions decisions. 


Though data can help high school students choose the right college, there are several 
challenges involved in accurately estimating the extent to which the specific school a student 
attends makes causal contributions to student success. One important limitation of Federal 
data sources is the lack of individual student-level data indicating academic preparation for 
college, such as their high-school GPA or college admissions test scores (e.g., SAT or ACT 
scores). Since academic preparation is an important element that adds context to measures of 
college quality, omitting this variable may bias estimates of college quality. As the College 
Scorecard continues to be refined and developed, these are challenges that the Department of 
Education will continue to face. 


In making admissions decisions, institutions of higher education may use big data techniques to 
try to predict the likelihood that an applicant will graduate before they ever set foot on 
campus.°? Using these types of data practices, some students could face barriers to admission 
because they are statistically less likely to graduate. Institutions could also deny students from 
low-income families, or other students who face unique challenges in graduating, the financial 
support that they deserve or need to afford college. This, in turn, creates a concern that as 
schools rush to cut costs, some applicants might face greater barriers to admission if they are 
considered unworthy of the extra resources it would take to keep them enrolled.°° One 
significant predictor of whether or not a student will graduate from college is family income, 
and the use of big data in this case may discriminate against students from lower-income 
families.°* The same data used to help students succeed can also be used to discourage low- 
income students from enrolling, and while there may be ways to mitigate this, such as financial 
incentives, especially at less selective institutions, it remains a cautionary example of using data 
to perpetuate discrimination. 


On the other hand, some schools and states are actively using data to promote access and 
success, and to prevent discrimination. For example, the State of Tennessee’s outcomes-based 
funding formula for four-year institutions offers an illustration of how data can promote both 
student success and access.°* Tennessee’s model places extra value on the “credit 
accumulation” and “degree attainment” outcomes of both students eligible for Pell grant 
funding and adult students (those over the age of 24).° In particular, these outcomes are 
valued 40 percent more than the same outcomes for non-Pell grant eligible traditional-age 
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students.°* By doing this, institutions have an incentive to enroll and promote the success of 
lower-income and adult students, who are traditionally under-represented in higher 
education.°° 


There are valuable opportunities for the use of big data in higher education, but they must be 
implemented with care. As learning itself is a process of trial and error, it is particularly 
important to use data in a manner that allows the benefits of those innovations, but still allows 
a safe space for students to explore, make mistakes, and learn without concern that there will 
be long term consequences for errors that are part of the learning process. 


Big Data and Criminal Justice 


The Problem: In a rapidly evolving world, law enforcement officials are looking for smart 
ways to use new technologies to increase community safety and trust. 


Local, state, and federal law enforcement agencies are increasingly drawing on data analytics 
and algorithmic systems to further their mission of protecting America. Using information 
gathered from the field and through the use of new technologies, law enforcement officials are 
analyzing situations in order to determine the appropriate response. At the same time, law 
enforcement agencies are expected to be accountable at all times to the communities they 
serve and will continue to be so in the digital age. Similarly, the technologies that assist law 
enforcement’s decisions and actions should also be accountable to ensure they are used ina 
thoughtful manner that considers the impact on communities and promotes successful 
community partnerships built on trust. 


The Big Data Opportunity: Data and algorithms can potentially help law enforcement 
become more transparent, effective, and efficient. 


Law enforcement agencies have long attempted to identify patterns in criminal activity in order 
to allocate scarce resources more efficiently. New technologies are replacing manual 
techniques, and many police departments now use sophisticated computer modeling systems 
to refine their understanding of crime hot spots, linking offense data to patterns in 
temperature, time of day, proximity to other structures and facilities, and other variables. The 
President’s Task Force on 21* Century Policing recommended, among many other steps, that 
law enforcement agencies adopt model policies and best practices for technology-based 
engagement that increases community trust and access; work toward national standards on the 
issue of technology’s impact on privacy concerns; and develop best practices that can be 
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adopted by state legislative bodies to govern the acquisition, use, retention, and dissemination 
of auditory, visual, and biometric data by law enforcement.® 


Since the Task Force released its recommendations, the White House and the Department of 
Justice have been engaged in several initiatives to ensure that the report’s recommendations 
are put into practice across the United States. As part of these efforts, the White House 
launched the Police Data Initiative to make policing data more transparent and improve 
community trust.® More than 50 police departments throughout the nation have joined in this 
work to realize the benefits of better technology. Commitments from participating jurisdictions 
include: increased use of open policing data to build community trust and increase 
departmental transparency, and use of data to more effectively identify policies that could be 
improved or officers who may contribute to adverse public interactions so they can be linked 
with effective training and interventions.°° 


Consistent with these goals, several police departments in the United States have developed 
and deployed “early warning systems” to identify officers who may benefit from additional 
training, resources, or counseling to prevent excessive uses of force, citizen complaints and 
other problems.®? Using de-identified police data, as well as contextual data about local crime 
and demographics, these systems are designed to detect the factors most indicative of future 
problems by attempting to determine behavioral patterns that predict a higher risk of future 
adverse incidents. Detecting these patterns opens new opportunities to develop targeted 
interventions for officers to protect their safety and improve police/community interactions. 


Separately, some of the newest analytical modeling techniques, often called “predictive 
policing,” might provide greater precision in predicting locations and times at which criminal 
activity is likely to occur. Research demonstrates that a neighborhood that has recently been 
victimized by one or more burglaries is likely to be targeted for additional property crimes in 
the coming days. An analytical method known as “near-repeat modeling” attempts to predict 
crimes based on this insight.” Similarly, a technique known as “risk terrain modeling” can 
identify specific locations where criminal activity often clusters, such as bars, motels or 
convenience stores, and can predict the specific social and physical factors that attract would- 
be offenders and create conditions ripe for criminal activity.’ Current Los Angeles Police 
Department (LAPD) Chief of Police, Charlie Beck, described predictive policing as enabling 
“directed, information-based patrol; rapid response supported by fact-based prepositioning of 
assets; and proactive, intelligence-based tactics, strategy, and policy.”’2 In some instances these 
systems have shown significant promise. In experiments conducted by the LAPD’s Foothill 
Division in which large sets of policing data were analyzed to predict occurrences of crime, the 
Division experienced a larger reduction in reported crime than any other division in the 
Department.” 
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The Big Data Challenge: The law enforcement community can use new technologies to 
enhance trust and public safety in the community, especially through measures that 
promote transparency and accountability and mitigate risks of disparities in treatment 
and outcomes based on individual characteristics. 


When designed and deployed carefully, data-based methodologies can help law enforcement 
make decisions based on factors and variables that empirically correlate with risk, rather than 
on flawed human instincts and prejudices. However, it is important that data and algorithmic 
systems not be used in ways that exacerbate unwarranted disparities in the criminal justice 
system. For example, unadjusted data could entrench rather than ameliorate documented 
racial disparities where they already exist, such as in traffic stops and drug arrest rates.”4 


Those leading efforts to use data analytics to create and implement predictive tools must work 
hard to ensure that such algorithms are not dependent on factors that disproportionately single 
out particular communities based on characteristics such as race, religion, income level, 
education, or other data inputs that may serve as proxies for characteristics with little or no 
bearing on an individual’s likelihood of association with criminal activity. For instance, when 
historical information is used with predictive algorithms to direct patrols, prior arrest data could 
be used to advise beat officers to patrol certain areas with greater frequency or intensity. If 
feedback loops are not thoughtfully constructed, a predictive algorithmic system built in this 
manner could perpetuate policing practices that are not sufficiently attuned to community 
needs and potentially impede efforts to improve community trust and safety. For example, 
machine learning systems that take into account past arrests could indicate that certain 
communities require more policing and oversight, when in fact the communities may be 
changing for the better over time. Moving forward, law enforcement agencies could work to 
account for these issues: transparency and accountability on data input and processes, a focus 
on eliminating data that could serve as proxies for race or poverty, and ensuring that bias is not 
replicated through these tools are key steps. 


It is also important to note that criminal justice data is notoriously poor.” This is in part 
because one of the major data repositories, the Federal Bureau of Investigation’s Uniform 
Crime Report (UCR), is in need of modernization and relies on voluntary contributions that 
often do not capture data with the degree of richness and completeness needed for in-depth 
analysis. FBI Director James Comey has prioritized improving data collection and repeatedly 
called on communities across the United States to increase participation in the UCR’s National 
Incident-Based Reporting System (NIBRS), explaining that this method of collecting data 
enhances our understanding of crime because “[it] doesn’t just include statistics. It gives the full 
picture—the circumstances and the context involving each incident. It asks: What happened? 
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Where did it happen? What time did it occur? Who was there and what is their demographic 
information? What is the relationship between the perpetrator and the victim?” ”° 


Even if crime reporting is improved, there will remain reasons to approach any crime dataset 
with care and caution. Many criminal-justice data inputs are inherently subjective. Officers use 
discretion in enforcement decisions (e.g., deciding whom to stop, search, question, and arrest) 
just as police officers and prosecutors use discretion in charging (e.g., simple assault vs. 
felonious assault). The underlying data reflects these judgement calls. 


Policymakers should also continue to look for ways to better use the increasing amount of data 
that law enforcement agencies now have in order to improve public safety and accountability. 
For example, the number of agencies presently using body cameras is expanding exponentially. 
With this expansion comes thousands of hours of video and audio. As part of the Police Data 
Initiative, the White House has engaged with academics and technologists to determine if there 
is a machine-readable way to review this video and audio to identify both beneficial and 
problematic interactions between law enforcement and civilian community members. On 
December 8, 2015, the White House Office of Science and Technology Policy participated in a 
workshop co-hosted by Stanford University and the City of Oakland Police Department focused 
on accelerating research and development to make body-worn camera data more searchable 
and interoperable with other systems, and on automating processes to reduce reporting 
burdens. 


More broadly, the conversation about ways to effectively use predictive analytics in law 
enforcement should continue, building on the work that has already begun between key 
stakeholders—ranging from law enforcement agencies to academics, community leaders and 
civil society groups. 


LOOKING TO THE FUTURE 





The use of big data can create great value for the American people, but as these technologies 
expand in reach throughout society, we must uphold our fundamental values so these systems 
are neither destructive nor opportunity limiting. Moving forward, it is essential that the public 
and private sectors continue to have collaborative conversations about how to achieve the 
most out of big data technologies while deliberately applying these tools to avoid—and when 
appropriate, address—discrimination. In order to ensure growth in the use of data analytics is 
matched with equal innovation to protect the rights of Americans, it will be important to: 


e Support research into mitigating algorithmic discrimination, building systems that 
support fairness and accountability, and developing strong data ethics frameworks. 


The Networking and Information Technology Research and Development Program and 
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the National Science Foundation (NSF) are developing research strategy proposals that 
will incorporate these elements and encourage researchers to continue to look at these 
issues. Through its support of the Council for Big Data, Ethics, and Society, as well as 
other efforts, NSF will continue to work with scientific, technical, and academic leaders 
to encourage the inclusion of data ethics within both research projects and student 
coursework and to develop interdisciplinary frameworks to help researchers, 
practitioners, and the public understand the complex issues surrounding big data, 
including discrimination, disparate impact, and associated issues of transparency and 
accountability. In particular, it will be important to bring together computer scientists, 
social scientists, and those studying the humanities in order to understand these issues 
in their historical, social, and technological contexts.” 


Encourage market participants to design the best algorithmic systems, including 
transparency and accountability mechanisms such as the ability for subjects to correct 


inaccurate data and appeal algorithmic-based decisions. Big data technologies can 
support the success of public and private institutions, but to do so, they must be 
implemented in a responsible and ethical manner. Organizations, institutions, and 
companies should be held accountable for the decisions they make with the aid of 
computerized decision-making systems and technology. The FTC’s recent big data report 
included considerations for companies using big data techniques, discussed potentially 
applicable laws, and suggested questions for legal compliance.’® Private companies 
using data analytics to expand opportunity should take these considerations into 
account in order to ensure that they treat consumers, students, job candidates, and the 
public fairly. Both private and public entities should also consider improved methods of 
providing individuals and communities with the means to access and correct their data, 
as well as better ways of providing notice about how their information is being used to 
inform decisions, such as those described in the case studies of this report. Recognizing 
the research issues outlined, experts from the data science and social science 
communities, among others, should continue to develop additional best practices for 
fair and ethical use of big data techniques and machine learning in the public and 
private sectors. 


Promote academic research and industry development of algorithmic auditing and 
external testing of big data systems to ensure that people are being treated fairly. 
One way these issues can be tackled is through the emerging field of algorithmic 
systems accountability, where stakeholders and designers of technology “investigate 
normatively significant instances of discrimination involving computer algorithms” and 
use nascent tools and approaches to proactively avoid discrimination through the use of 
new technologies employing research-based behavior science. ’? These efforts should 
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also include an analysis identifying the constituent elements of transparency and 
accountability to better inform the ethical and policy considerations of big data 
technologies. There are other promising avenues for research and development that 
could address fairness and discrimination in algorithmic systems, such as those that 
would enable the design of machine learning systems that constrain disparate impact or 
construction of algorithms that incorporate fairness properties into their design and 
execution. 


Broaden participation in computer science and data science, including opportunities to 
improve basic fluencies and capabilities of all Americans. Consistent with the goals of 


the President’s Computer Science for All and TechHire initiatives, educational 
institutions and employers can strive to broaden participation in these fields.®° In 
particular, they should look for ways to provide more Americans with opportunities to 
have greater fluency and awareness of how these issues impact them and to influence 
how these fields evolve going forward. 


Consider the roles of the government and private sector in setting the rules of the road 
for how data is used. As use of big data moves from new and novel to mainstream, the 
private sector, citizens, institutions, and the public sector are establishing expectations, 
norms, and standards that will serve as guides for the future. How big data is used 
ethically to reduce discrimination and advance opportunity, fairness, and inclusion 
should inform the development of both private sector standards and public policy 
making in this space. 
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